Configure environment versions for pipelines

Important

Environment versions for Lakeflow pipelines are in Public Preview.

An environment version pins the Python language version and the set of preinstalled Python libraries available to your pipeline's Python code. Any external dependencies you add to the pipeline are layered on top of this base.

Environment versions decouple your pipeline's Python runtime from the Databricks Runtime version your pipeline runs on. While an environment version is set, Databricks Runtime upgrades don't change your Python language version or preinstalled library versions. The Python runtime is also consistent with serverless Jobs and notebooks that use the same environment version. To find the current Databricks Runtime version for Lakeflow pipelines, see Lakeflow pipelines release notes and the release upgrade process.

Databricks automatically migrates eligible pipelines onto an environment version. See Automatic migration.

Important

Pipelines with an environment version run Python code through Spark Connect, which can change the behavior of some pipeline code. For the full list of limitations and behavior changes, see Environment version compatibility.

Requirements

Environment versions have the following requirements:

  • The pipeline must use Unity Catalog. Hive metastore pipelines are not supported.

Supported environment versions

Lakeflow pipelines support environment versions 3 and 4 on both serverless and classic compute. For the Python language version and the full list of preinstalled Python libraries available in each version, see the environment version reference.

Automatic migration

Databricks automatically migrates eligible pipelines onto an environment version. Migration happens on a pipeline's next update, requires no manual steps, and includes these safeguards:

  • Behavior is checked before migration. Databricks scans your pipeline for code patterns that would behave differently under Spark Connect and compares the pipeline's output plan before and after migration. See Environment version compatibility.
  • A failed migration reverts automatically. If a migrated update fails for any reason, it is stopped before any data is written, and the pipeline automatically reverts to its previous runtime on the next update. No action is required.
  • Your explicit settings are always respected. If you set environment_version yourself, Databricks never overrides it.

Once a pipeline migrates successfully, it runs on the environment version going forward. You can see the version that was selected in the pipeline settings, the GetPipeline API response, and the runtime_details event log.

What is not migrated automatically

A pipeline is not migrated automatically if any of the following apply:

Pipelines that are not migrated automatically keep running on their previous Python runtime with no change. You can enable an environment version yourself once the pipeline is eligible.

Enable an environment version yourself

Automatic migration covers most pipelines. You can also configure an environment version explicitly, for example to migrate sooner or to pin a specific version, through the pipeline editor UI, the Pipelines REST API, or Declarative Automation Bundles.

Before you enable an environment version explicitly, check that your pipeline is compatible with Spark Connect. If Databricks detects that enabling the version would change your pipeline's behavior, the update fails before any data is written, with an error identifying the difference to resolve.

Enable through the UI

  1. From the pipeline editor, click Settings.
  2. Under Pipeline Environment, select Pencil icon. Edit environment.
  3. Select an environment version from the dropdown list.
  4. Save the pipeline settings.

External dependencies added in the Pipeline Environment section are layered on top of the libraries included with the selected environment version. See Manage Python dependencies for pipelines.

Enable through the API

The Pipelines REST API accepts an environment block on pipeline create and update. Personal Access Token authentication must be enabled for the workspace.

To create a pipeline with an environment version:

curl --request POST \
  --url 'https://<workspace-host>/api/2.0/pipelines' \
  --header 'Authorization: Bearer <personal-access-token>' \
  --header 'Content-Type: application/json' \
  --data-raw '{
    "name": "<pipeline-name>",
    "catalog": "<catalog>",
    "schema": "<schema>",
    "channel": "CURRENT",
    "environment": {
      "environment_version": "4",
      "dependencies": [
        "simplejson==3.19.*"
      ]
    }
  }'

To set the environment version on an existing pipeline, send the same environment block with PUT /api/2.0/pipelines/<pipeline-id>.

Enable through Declarative Automation Bundles

When you create a pipeline using Declarative Automation Bundles, you can set an environment version in the YAML definition of the pipeline.

  1. Make sure your Databricks CLI is at version v0.294.0 or later. If not, upgrade by following the installation guide.
  2. Set up a bundle by following the pipelines bundle tutorial.
  3. Locate the pipeline YAML in your bundle, typically <bundle-folder>/resources/<pipeline_name>_pipeline.yml.
  4. Set the environment_version and dependencies fields in the pipeline YAML:
resources:
  pipelines:
    my_pipeline:
      name: my_pipeline
      catalog: ${var.catalog}
      schema: ${var.schema}
      root_path: '../src/my_pipeline'
      libraries:
        - glob:
            include: ../src/my_pipeline/transformations/**
      environment:
        environment_version: 4
        dependencies:
          - --editable ${workspace.file_path}

Check the environment version on a pipeline

To check which environment version a pipeline is running, whether you set it explicitly or Databricks selected it during automatic migration:

  • UI: Open the pipeline settings and check the Pipeline Environment section, or inspect the JSON pane for the environment.environment_version field.
  • API: Call GET /api/2.0/pipelines/<pipeline-id> and look for environment.environment_version in the response. This shows a version you set explicitly; to see a version selected by automatic migration, use the event log below.
  • Event log: Inspect the runtime_details event, which reports the environment version used for the update. See Details for runtime_details event.

Disable or roll back an environment version

You don't need to roll back an automatic migration manually. A migration that fails is reverted automatically on the next update. When the environment version is removed, the pipeline returns to its previous Python runtime configuration. If anything unexpected happens, contact Azure Databricks support.

Additional resources

  • Last updated on