Design and implement data pipelines with Azure Databricks

Module
10 Units

Intermediate

Data Engineer

Azure Databricks

Learn to design and implement robust data pipelines in Azure Databricks using notebooks and Lakeflow Spark Declarative Pipelines, covering orchestration, error handling, and task logic.

Learning objectives

By the end of this module, you'll be able to:

Design the order of operations for data pipelines from ingestion to serving.
Choose between notebooks and Lakeflow Spark Declarative Pipelines based on use cases.
Design task logic, dependencies, and execution patterns for Lakeflow Jobs.
Implement error handling strategies, including retry policies and expectations.
Create data pipelines using both notebooks and Lakeflow Spark Declarative Pipelines.

Prerequisites

The following prerequisites should be completed:

Basic understanding of Azure Databricks workspaces
Familiarity with Unity Catalog fundamentals and governance concepts
Knowledge of SQL and data organization principles

Introduction min
Design order of operations for a pipeline min
Choose notebook vs Lakeflow Pipelines min
Design Lakeflow job logic min
Design error handling in pipelines and jobs min
Create pipeline with notebook min
Create pipeline with Lakeflow Spark Declarative Pipelines min
Exercise - Design and Implement Data Pipelines with Azure Databricks min
Module assessment min
Summary min

Start