Pipeline developer reference

Implement data loading and transformations in Lakeflow Spark Declarative Pipelines with queries that define streaming tables and materialized views. Lakeflow Spark Declarative Pipelines supports both SQL and Python interfaces. Because they provide equivalent functionality for most data processing use cases, you can choose whichever interface you are most comfortable with.

Python development

Create pipelines using Python code.

Topic	Description
Develop pipeline code with Python	An overview of developing pipelines in Python.
Lakeflow Spark Declarative Pipelines Python language reference	Python reference documentation for the pipelines module.
Manage Python dependencies for pipelines	Instructions for managing Python libraries in pipelines.
Import Python modules from Git folders or workspace files	Instructions for using Python modules that you have stored in Azure Databricks.

SQL development

Create pipelines using SQL code.

Topic	Description
Develop Lakeflow Spark Declarative Pipelines code with SQL	An overview of developing pipelines in SQL.
Pipeline SQL language reference	Reference documentation for SQL syntax for Lakeflow Spark Declarative Pipelines.
Standalone pipelines	Use Databricks SQL to work with pipelines.

Other development topics

The following topics describe other ways to develop pipelines.

Topic	Description
Convert a pipeline into a bundle project	Convert an existing pipeline to a bundle, which allows you to manage your data processing configuration in a source-controlled YAML file for easier maintenance and automated deployments to target environments.
Metaprogramming with Lakeflow Spark Declarative Pipelines	Create pipelines with dlt-meta. Use the open source dlt-meta library to automate the creation of pipelines with a metadata-driven framework. Tutorial: Create multiple flows with different parameters. Create multiple flows in a loop in Python.
Develop pipeline code in your local development environment	An overview of options for developing pipelines locally.