Lakeflow pipelines release notes and the release upgrade process

This article explains the Lakeflow pipelines release process, how the runtime is managed, and provides links to release notes for each release.

Current Databricks Runtime channel versions

The current Databricks Runtime versions for each channel are:

For the list of packages installed in each runtime — Python, R, and Java libraries — see the corresponding Databricks Runtime release notes linked above.

Lakeflow pipelines release notes

Release notes are organized by month. Because pipelines are versionless, both workspace and runtime changes take place automatically. The following release notes provide an overview of changes and bug fixes in each release:

Previously, release notes were organized by year and week-of-year.

Older release notes

Lakeflow pipelines runtime channels

Pipeline clusters use runtimes based on Databricks Runtime release notes versions and compatibility. Azure Databricks automatically upgrades the pipeline runtimes to support enhancements and upgrades to the platform. You can use the channel field in the pipeline settings to control the runtime version that runs your pipeline. The supported values are:

  • current to use the current runtime version.
  • preview to test your pipeline with upcoming changes to the runtime version.

By default, your pipelines run using the current runtime version. Databricks recommends using the current runtime for production workloads. To learn how to use the preview setting to test your pipelines with the next runtime version, see Automate testing of your pipelines with the next runtime version.

Important

Features marked as generally available or Public Preview are available in the current channel.

For more information about pipeline channels, see the channel field in the pipeline settings.

To understand how upgrades are managed for each release, see How do Lakeflow pipeline upgrades work?.

How do I find the Databricks Runtime version for a pipeline update?

You can query the pipeline event log to find the Databricks Runtime version for a specific pipeline update. See Runtime information.

How do Lakeflow pipeline upgrades work?

Lakeflow pipelines are versionless, which means that Azure Databricks automatically upgrades the runtime to support enhancements and upgrades to the platform. Databricks recommends limiting external dependencies.

Databricks proactively works to prevent automatic upgrades from introducing errors or issues to production pipelines. See Lakeflow pipelines upgrade process.

Especially for users that deploy pipelines with external dependencies, Databricks recommends proactively testing pipelines with preview channels. See Automate testing of your pipelines with the next runtime version.

Lakeflow pipelines upgrade process

Azure Databricks manages the Databricks Runtime used by pipeline compute resources. It automatically upgrades the runtime in your Azure Databricks workspaces and monitors the health of your pipelines after the upgrade.

If Azure Databricks detects that a pipeline cannot start because of an upgrade, the runtime version for the pipeline reverts to the previous version that is known to be stable, and the following steps are triggered automatically:

  • The pipeline's runtime is pinned to the previous known-good version.
  • Databricks support is notified of the issue.
    • If the issue is related to a regression in the runtime, Databricks resolves the issue.
    • If the issue is caused by a custom library or package used by the pipeline, Databricks contacts you to resolve the issue.
  • When the issue is resolved, Azure Databricks initiates the upgrade again.

Important

Azure Databricks only reverts pipelines running in production mode with the channel set to current.

Automate testing of your pipelines with the next runtime version

To ensure changes in the next runtime version do not impact your pipelines, use the pipeline channels feature:

  1. Create a staging pipeline and set the channel to preview.
  2. In the pipeline UI, create a schedule to run the pipeline weekly and enable alerts to receive an email notification for pipeline failures. Databricks recommends scheduling weekly test runs of pipelines, especially if you use custom pipeline dependencies.
  3. If you receive a notification of a failure and are unable to resolve it, open a support ticket with Databricks.

Note

If you modify your pipeline while resolving preview channel failures, validate those changes against the current channel before deploying to production running on current. A change that works on preview might behave differently on current.

Pipeline dependencies

Pipelines support external dependencies; for example, you can install any Python package using the %pip install command. They also support using global and cluster-scoped init scripts. However, these external dependencies, particularly init scripts, increase the risk of issues with runtime upgrades. To mitigate these risks, minimize using init scripts in your pipelines. If your processing requires init scripts, automate testing of your pipeline to detect problems early; see Automate testing of your pipelines with the next runtime version. If you use init scripts, Databricks recommends increasing your testing frequency.

  • Last updated on