Choose a batch processing technology in Azure
Big data solutions often use long-running batch jobs to filter, aggregate and otherwise prepare the data for analysis. Usually, these jobs involve reading source files from scalable storage (like HDFS, Azure Data Lake Store, and Azure Storage), processing them, and writing the output to new files in scalable storage.
The fundamental requirement of such batch processing engines is to scale out computations to handle a large volume of data. Unlike real-time processing, batch processing is expected to have latencies (the time between data ingestion and computing a result) that measure in minutes to hours.
Technology choices for batch processing
Azure Synapse Analytics
Azure Synapse is a distributed system designed to perform analytics on large data. It supports massive parallel processing (MPP), which makes it suitable for running high-performance analytics. Consider Azure Synapse when you have large amounts of data (more than 1 TB) and are running an analytics workload that will benefit from parallelism.
Azure Data Lake Analytics
Data Lake Analytics is an on-demand analytics job service. It's optimized for distributed processing of large data sets stored in Azure Data Lake Store.
- Languages: U-SQL (including Python, R, and C# extensions).
- Integrates with Azure Data Lake Store, Azure Storage blobs, Azure SQL Database, and Azure Synapse.
- Pricing model is per-job.
- Languages: R, Python, Java, Scala, SQL
- Kerberos authentication with Active Directory, Apache Ranger-based access control
- Gives you complete control of the Hadoop cluster
Azure Databricks is an Apache Spark-based analytics platform. You can think of it as "Spark as a service." It's the easiest way to use Spark on the Azure platform.
- Languages: R, Python, Java, Scala, Spark SQL
- Fast cluster start times, autotermination, autoscaling.
- Manages the Spark cluster for you.
- Built-in integration with Azure Blob Storage, Azure Data Lake Storage (ADLS), Azure Synapse, and other services. See Data Sources.
- User authentication with Azure Active Directory.
- Web-based notebooks for collaboration and data exploration.
- Supports GPU-enabled clusters
Key selection criteria
To narrow the choices, start by answering these questions:
Do you want a managed service rather than managing your own servers?
Do you want to author batch processing logic declaratively or imperatively?
Will you perform batch processing in bursts? If yes, consider options that let you auto-terminate the cluster or whose pricing model is per batch job.
Do you need to query relational data stores along with your batch processing, for example, to look up reference data? If yes, consider the options that enable the querying of external relational stores.
The following tables summarize the key differences in capabilities.
|Capability||Azure Data Lake Analytics||Azure Synapse||HDInsight||Azure Databricks|
|Is managed service||Yes||Yes||Yes 1||Yes|
|Relational data store||Yes||Yes||No||No|
|Pricing model||Per batch job||By cluster hour||By cluster hour||Databricks Unit2 + cluster hour|
 With manual configuration.
 A Databricks Unit (DBU) is a unit of processing capability per hour.
|Capability||Azure Data Lake Analytics||Azure Synapse||HDInsight with Spark||HDInsight with Hive||HDInsight with Hive LLAP||Azure Databricks|
|Scale-out granularity||Per job||Per cluster||Per cluster||Per cluster||Per cluster||Per cluster|
|In-memory caching of data||No||Yes||Yes||No||Yes||Yes|
|Query from external relational stores||Yes||No||Yes||No||No||Yes|
|Authentication||Azure AD||SQL / Azure AD||No||Azure AD1||Azure AD1||Azure AD|
|Auditing||Yes||Yes||No||Yes 1||Yes 1||Yes|
|Row-level security||No||Yes2||No||Yes 1||Yes 1||No|
|Supports firewalls||Yes||Yes||Yes||Yes 3||Yes 3||No|
|Dynamic data masking||No||Yes||No||Yes 1||Yes 1||No|
 Requires using a domain-joined HDInsight cluster.
 Filter predicates only. See Row-Level Security
 Supported when used within an Azure Virtual Network.
This article is maintained by Microsoft. It was originally written by the following contributors.
- Zoiner Tejada | CEO and Architect
- Create a lake database in Azure Synapse Analytics
- Create an Azure Databricks workspace
- Explore Azure Databricks
- Get started with Azure Data Lake Analytics using the Azure portal
- Introduction to Azure Synapse Analytics
- What is Azure Databricks?
- What is Azure Synapse Analytics?