Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
An extract, transform, and load (ETL) pipeline turns the raw data that you ingest into clean, validated tables that analysts, dashboards, and models can trust. On Azure Databricks, you build ETL pipelines with Lakeflow pipelines. You declare the tables that the pipeline produces in SQL or Python, and the pipeline works out the order to update them, processes only new or changed data when possible, and automatically retries recoverable failures in scheduled runs.
Note
Lakeflow pipelines require the Premium plan. Contact your Databricks account team for more information.
Where pipelines fit in Lakeflow
Lakeflow is the Azure Databricks data engineering solution for ingesting, transforming, and orchestrating data. A typical workflow uses all three of its components:
| Component | Role | Instructions |
|---|---|---|
| Lakeflow Connect | Ingests data from databases, software as a service (SaaS) applications, files, and streaming sources into Unity Catalog tables | Ingest data from databases and SaaS applications |
| Lakeflow pipelines | Transforms ingested data into cleaned and aggregated tables with streaming and batch processing | This section |
| Lakeflow Jobs | Orchestrates pipelines, notebooks, and other tasks on a schedule or in response to a trigger | Orchestrate workloads with Lakeflow Jobs |
The following videos introduce Lakeflow and show it in use:
How Lakeflow pipelines work
Lakeflow pipelines are built on Apache Spark⢠Declarative Pipelines (SDP), and run on the performance-optimized Databricks Runtime. Code that uses only the features that SDP shares, such as streaming tables, materialized views, and flows, stays portable to other SDP runtimes. Production features such as change data capture (CDC) with AUTO CDC and data quality expectations are specific to Lakeflow pipelines. See Apache Spark Declarative Pipelines.
A pipeline is made up of the following components:
- Streaming tables: Delta tables that process data incrementally. Each run reads only the data that arrived since the previous run, so streaming tables suit append-heavy sources such as files in cloud storage. See Streaming tables.
- Materialized views: Views with cached results that the pipeline keeps up to date. Materialized views suit aggregations and joins, and the pipeline refreshes them incrementally when possible. See Materialized views.
- Flows: The queries that read from a source and write to a streaming table or a materialized view. See Load and process data incrementally with Lakeflow pipeline flows.
- Expectations: Data quality constraints on a table. Each expectation defines what happens to records that fail it: keep them and record the failure, drop them, or stop the update. See Manage data quality with pipeline expectations.
The pipeline infers its execution graph from the tables that your code references, so you don't write orchestration, checkpointing, or retry logic. For how you use pipelines from design through production, see How to use Spark Declarative Pipelines on Lakeflow.
Organize tables in bronze, silver, and gold layers
Databricks recommends the medallion architecture, which organizes pipeline tables into layers that progressively improve the quality of the data:
- Bronze: Raw data, ingested as is from the source.
- Silver: Cleaned and validated data, with expectations that filter or flag bad records.
- Gold: Aggregated and enriched data, modeled for analytics and AI.
See What is the medallion lakehouse architecture?.
Build your first pipeline
Choose one of the following approaches:
| Approach | Description | Instructions |
|---|---|---|
| Install a demo | Install a complete bronze, silver, and gold pipeline with sample data and dashboards, and then explore how it works | Install a sample ETL pipeline with dbdemos |
| Use Genie Code | Describe the pipeline in natural language in the Lakeflow Pipelines Editor, and Genie Code writes and runs the pipeline code | Build a pipeline with Genie Code |
The rest of the Starter Journey uses the Genie Code path. That path builds bakehouse_e2e_pipeline, which reads the samples.bakehouse sample data and writes bronze tables to bakehouse_bronze and silver tables to bakehouse_silver, in starterjourney_catalog. The query and Genie Ontology pages depend on the silver table and column names that this pipeline produces. The demo installs its own tables, so those pages can't find them. If you start with the demo, build the Bakehouse pipeline with Genie Code before you continue to the query page.
To write the pipeline code yourself, see Lakeflow pipelines tutorials. For the language references, see Pipeline SQL language reference and Lakeflow pipelines Python language reference.
Next steps
| Resource | Description |
|---|---|
| Install a sample ETL pipeline with dbdemos | Install a sample bronze, silver, and gold pipeline, and explore its tables, expectations, and dashboards |
| Build a pipeline with Genie Code | Build a pipeline from a natural language description in the Lakeflow Pipelines Editor |