Develop pipeline code in your local development environment

You can author Python pipeline source code in your preferred integrated development environment (IDE), run it locally for testing, then validate, deploy, and run updates in your Azure Databricks workspace without leaving your local environment.

Lakeflow pipelines are a superset of Apache Spark™ Declarative Pipelines. Code that uses only Apache Spark Declarative Pipelines APIs runs both locally and on Azure Databricks, but code that uses features unique to Lakeflow pipelines, such as AUTO CDC and expectations, runs only on Azure Databricks. For the feature differences, see Lakeflow pipelines Python language reference.

For interactive development and testing in the Azure Databricks workspace, use the Lakeflow Pipelines Editor. See Develop and debug ETL pipelines with the Lakeflow Pipelines Editor.

Write pipeline code with IDE support

Write pipeline code using the pyspark.pipelines module, imported as dp:

from pyspark import pipelines as dp

Because the module is part of Apache Spark, your IDE provides syntax checking, autocomplete, and type checking as you write. Apache Spark Declarative Pipelines code typically runs without modification on Azure Databricks. Running the same import command in a Lakeflow pipeline imports the Azure Databricks version of pipelines. For the full Lakeflow pipelines Python reference, see Lakeflow pipelines Python language reference.

Separate transformation logic for local testing

The most effective way to make pipeline code testable locally is to keep your transformation logic in plain PySpark functions, separate from the dp decorators. A function that takes a DataFrame and returns a DataFrame has no dependency on the Lakeflow pipelines runtime, so you can unit test it with pytest on your local machine, just like any other Apache Spark code. Keep the decorated functions thin, so they import the logic and call it:

# transformations/clean.py — pure PySpark, unit-testable on its own
def clean_orders(df):
    return df.filter("quantity > 0").withColumn("amount_usd", df.amount.cast("double"))

# pipeline file — a thin dp wrapper that imports and calls the logic
from pyspark import pipelines as dp
from transformations.clean import clean_orders

@dp.table(name="orders_silver")
def orders_silver():
    return clean_orders(spark.readStream.table("orders_bronze"))

You can package the shared logic as a wheel to reuse it across pipelines. For a full walkthrough of writing and running unit tests, see Unit testing for pipelines.

Run pipelines locally for testing

You can also run pipelines locally to develop and test your code before you run it on Azure Databricks. Use the spark-pipelines command line interface to initialize, validate, and run a pipeline with local Apache Spark. See the Spark Declarative Pipelines Programming Guide in the Apache Spark documentation.

A complete pipeline uses three complementary layers of testing, and you can exercise two of them locally:

• Unit tests for your transformation logic, run with pytest against the plain PySpark functions described above. These require no pipeline runtime. See Unit testing for pipelines.
• Validation (dry run) of the pipeline graph, source code, and dataset references, using spark-pipelines locally or databricks pipelines dry-run against your workspace—without writing any data.
• Expectations, which evaluate data quality rules on every row of every run. Because they are a Lakeflow pipelines runtime feature, they run only on Azure Databricks, not locally. See Manage data quality with pipeline expectations.

You cannot run or test functionality that is specific to Lakeflow pipelines locally. This includes expectations and AUTO CDC functions.

Run pipelines in Azure Databricks from your local environment

Use the databricks pipelines command group to validate, deploy, and run pipeline updates in your workspace, directly from your terminal:

databricks pipelines init      # scaffold a pipeline project
databricks pipelines dry-run   # validate the pipeline graph without publishing data
databricks pipelines deploy    # deploy the project to your workspace
databricks pipelines run       # run an update

Pipeline updates run in your Azure Databricks workspace, not on your local machine, using the compute configured for the pipeline. These commands are interoperable with Declarative Automation Bundles bundle commands, so you can start with a simple project and adopt bundle configuration and CI/CD practices as it grows. To install and configure the CLI, see Install or update the Databricks CLI. For the full command reference, see pipelines command group. For a step-by-step walkthrough, see Develop pipelines with Declarative Automation Bundles.

Sync pipeline code from your IDE to a workspace

The following table summarizes options for syncing pipeline source code between your local IDE and an Azure Databricks workspace:

Tool or pattern	Details
Databricks CLI (pipelines command group)	Use the databricks pipelines commands to deploy and run a pipeline project from your local environment. See pipelines command group.
Declarative Automation Bundles	Use Declarative Automation Bundles to deploy pipeline assets ranging in complexity from a single source code file to configurations for multiple pipelines, jobs, and source code files. See Convert a pipeline into a bundle project.
Databricks IDE extension	Azure Databricks provides an integration with Visual Studio Code that includes easy syncing between your local IDE and workspace files. This extension also provides tools for using Declarative Automation Bundles to deploy pipelines assets. See Databricks IDE extension.
Workspace files	You can use Databricks workspace files to upload your pipeline source code to your Databricks workspace and then import that code into a pipeline. See What are workspace files?.
Git folders	Git folders let you sync code between your local environment and Azure Databricks workspace using a Git repository as the intermediary. See Azure Databricks Git folders.