Differences between HD Insight and Azure Data bricks?

Question

I know that HDInsight has several types of clusters whereas Databricks is only for Spark type of cluster. I believe there must be some significant differences which will influence what to be chosen for implementation.

[Note: As we migrate from MSDN, this question has been posted by an Azure Cloud Engineer as a frequently asked question] Source: MSDN

Answer 1

Welcome to the Microsoft Q&A (Preview) platform.

Happy to answer your questions

Azure HDInsight is a cloud distribution of the Hadoop components from the Hortonworks Data Platform (HDP). Azure HDInsight makes it easy, fast, and cost-effective to process massive amounts of data. You can use the most popular open-source frameworks such as Hadoop, Spark, Hive, LLAP, Kafka, Storm, R, and more. For more details, refer to Azure HDInsight Documentation.

Azure HDInsight brings both Hadoop and Spark under the same umbrella and enables enterprises to manage both using the same set of tools e.g. using Ambari, Apache Ranger etc. It also offers industry standard notebook experience with support for both Jupyter and Zeppelin notebooks. Enterprises that want this ease of manageability across all their big data workloads can choose to use HDInsight.

Azure Databricks is a premium Spark offering that is ideal for customers who want their data scientists to collaborate easily and run their Spark based workloads efficiently and at industry leading performance.

Azure Databricks is an Apache Spark-based analytics platform optimized for the Microsoft Azure cloud services platform. For more details, refer to Azure Databricks Documentation.

Here is the comparison on Azure HDInsight vs Databricks.

For more details, refer MSDN thread which addressing similar question.

Hope this helps.

Sourced from MSDN – Azure HDInsight