Azure Databricks: End-to-end web-based analytics platform

What is Databricks and Azure Databricks?

Databricks is developed by the founders of Apache Spark and is an end-to end (from development to production) web-based analytics platform that makes it easy to combine Big Data, Data Science and Apacke Spark.

In 2017, Microsoft and Databricks, under the name Azure Databricks, entered into a collaboration that has enabled to fully integrate a Databricks platform in a Azure-environment.

This collaboration between Azure as a Cloud provider and Databricks as the Apache Spark platform, allows the huge computing power of Databricks to be integrated into a fully integrated cloud environment where the services speak the same language – now also with the Databricks framework

Want to know more?

Azure Databricks features

Databricks is basically created to make it easy to work with. The solution therefore contains a lot of user-friendly features:

Real-time visualizations in the Notebook

An internal dashboard connected to your Notebook

Integration with Github and Bitbucket

Direct scheduling of each Notebook

Databrick’s own Notebook that acts as Databrick’s own Interactive Development Environment (IDE)

Notebooks that support multiple languages: R, Python, SQL, and Scala

Azure Databricks and Apache Spark

One of the great strengths of the collaboration between Azure and Databricks is that you have an Apache Spark platform that is fully integrated with all known Azure components such as Azure Data Factory and Azure Blob Storage, allowing for continuous pipelines in each project.

A related important aspect of Databricks is the ability to share different profiles, making it easier and more secure for different profiles such, as Data Engineers and Data Scientist, to work together on individual projects in the Databricks environment.

Databricks also have the option of Auto-Scaling your resources. This means that you can have a cluster that automatically adapts to what you need at the given time. In general, the entire clustering aspect is handled by Databricks, which makes it easy to get started with Cluster Computing even for beginners. When you reach a more advanced level in the process of spark, there are also opportunities to monitor your programs directly from Databricks to optimize these.

How Databricks can be used

In addition to the general benefit of having an easy overview of your pipelines and cluster computing advantage when processing very large volumes of data, here we will look at two examples where Databrick’s makes particular sense:

Tasks that include using Machine Learning or Deep Learning

In addition to having an easy and clear integration to all the major Machine Learning libraries as well as necessities that come before and after Machine Learning development, Databricks has the advantage that the very large computations or data volumes that are commonly associated with Machine Learning and Deep Learning can easily be orchestrated on your automatically started cluster.

Tasks that include using real time streaming data such as IoT data from machines

Here you can take advantage of Databrick’s Apache Spark foundation, more specifically the extension to Apache Sparks core API: Spark Streaming, which is set up to work with frequently used data sources such as Kafka, HDFS and Twitter. Spark Streaming is also fully integrated into Databricks and is very easy to set up and use.