How does Azure Databricks manage Delta Lake feature compatibility?
Delta Lake is an independent open-source project under the governance of the Linux Foundation. Databricks introduces support for new Delta Lake features and optimizations that build on top of Delta Lake in Databricks Runtime releases.
Azure Databricks optimizations that leverage Delta Lake features respect the protocols used in OSS Delta Lake for compatability.
Many Azure Databricks optimizations require enabling Delta Lake features on a table. Delta Lake features are always backwards compatible, so tables written by a lower Databricks Runtime version can always be read and written by a higher Databricks Runtime version. Enabling some features breaks forward compatibility with workloads running in a lower Databricks Runtime version. For features that break forward compatibility, you must update all workloads that reference the upgraded tables to use a compliant Databricks Runtime version.
The following Delta Lake features break forward compatibility. Features are enabled on a table-by-table basis. This table lists the lowest Databricks Runtime version still supported by Azure Databricks.
|Feature||Requires Databricks Runtime version or later||Documentation|
||Databricks Runtime 9.1 LTS||Set a CHECK constraint in Azure Databricks|
|Change data feed||Databricks Runtime 9.1 LTS||Use Delta Lake change data feed on Azure Databricks|
|Generated columns||Databricks Runtime 9.1 LTS||Use Delta Lake generated columns|
|Column mapping||Databricks Runtime 10.4 LTS||Rename and drop columns with Delta Lake column mapping|
|Identity columns||Databricks Runtime 10.4 LTS||Use identity columns in Delta Lake|
|Table features||Databricks Runtime 12.1 (unsupported)||What are table features?|
|Deletion vectors||Databricks Runtime 12.1 (unsupported)||What are deletion vectors?|
|TimestampNTZ||Databricks Runtime 13.0||TIMESTAMP_NTZ type|
|Liquid clustering||Databricks Runtime 13.2||Use liquid clustering for Delta tables|
|UniForm||Databricks Runtime 13.2||Universal Format (UniForm) for Iceberg compatibility with Delta tables|
Delta Live Tables and Databricks SQL automatically upgrade runtime environments with regular releases to support new features. See Delta Live Tables release notes and the release upgrade process and Databricks SQL release notes.
Every Delta table has a protocol specification which indicates the set of features that the table supports. The protocol specification is used by applications that read or write the table to determine if they can handle all the features that the table supports. If an application does not know how to handle a feature that is listed as supported in the protocol of a table, then that application is not be able to read or write that table.
The protocol specification is separated into two components: the read protocol and the write protocol.
The read protocol lists all features that a table supports and that an application must understand in order to read the table correctly. Upgrading the read protocol of a table requires that all reader applications support the added features.
All applications that write to a Delta table must be able to construct a snapshot of the table. As such, workloads that write to Delta tables must respect both reader and writer protocol requirements.
If you encounter a protocol that is unsupported by a workload on Azure Databricks, you must upgrade to a higher Databricks Runtime that supports that protocol.
The write protocol lists all features that a table supports and that an application must understand in order to write to the table correctly. Upgrading the write protocol of a table requires that all writer applications support the added features. It does not affect read-only applications, unless the read protocol is also upgraded.
Which protocols must be upgraded?
Some features require upgrading both the read protocol and the write protocol. Other features only require upgrading the write protocol.
As an example, support for
CHECK constraints is a write protocol feature: only writing applications need to know about
CHECK constraints and enforce them.
In contrast, column mapping requires upgrading both the read and write protocols. Because the data is stored differently in the table, reader applications must understand column mapping so they can read the data correctly.
For more on upgrading, see Upgrading protocol versions.
In Databricks Runtime 12.1 and above, Delta Lake table features introduce granular flags specifying which features are supported by a given table. In Databricks Runtime 12.0 and below, Delta Lake features were enabled in bundles called protocol versions. Table features are the successor to protocol versions and are designed with the goal of improved flexibility for clients that read and write Delta Lake. See What is a protocol version?.
Table features have protocol version requirements. See Features by protocol version.
A Delta table feature is a marker that indicates that the table supports a particular feature. Every feature is either a write protocol feature (meaning it only upgrades the write protocol) or a read/write protocol feature (meaning both read and write protocols are upgraded to enable the feature).
To learn more about supported table features in Delta Lake, see the Delta Lake protocol.
Do table features change how Delta Lake features are enabled?
If you only interact with Delta tables through Azure Databricks, you can continue to track support for Delta Lake features using minimum Databricks Runtime requirements. Azure Databricks supports reading Delta tables that have been upgraded to table features in all Databricks Runtime LTS releases, as long as all features used by the table are supported by that release.
If you read and write from Delta tables using other systems, you might need to consider how table features impact compatibility, because there is a risk that the system could not understand the upgraded protocol versions.
Table features are introduced to the Delta Lake format for writer version 7 and reader version 3. Azure Databricks has backported code to all supported Databricks Runtime LTS versions to add support for table features, but only for those features already supported in that Databricks Runtime. This means that while you can opt in to using table features to enable generated columns and still work with these tables in Databricks Runtime 9.1 LTS, tables with identity columns enabled (which requires Databricks Runtime 10.4 LTS) are still not supported in that Databricks Runtime.
A protocol version is a protocol number that indicates a particular grouping of table features. In Databricks Runtime 12.0 and below, you cannot enable table features individually. Protocol versions bundle a group of features.
Delta tables specify a separate protocol version for read protocol and write protocol. The transaction log for a Delta table contains protocol versioning information that supports Delta Lake evolution. See Review Delta Lake table details with describe detail.
The protocol versions bundle all features from previous protocols. See Features by protocol version.
Starting with writer version 7 and reader version 3, Delta Lake has introduced the concept of table features. Using table features, you can now choose to only enable those features that are supported by other clients in your data ecosystem. See What are table features?.
The following table shows minimum protocol versions required for Delta Lake features.
If you are only concerned with Databricks Runtime compatibility, see What Delta Lake features require Databricks Runtime upgrades?. Delta Sharing only supports reading tables with features that require
|Basic functionality||2||1||What is Delta Lake?|
||3||1||Set a CHECK constraint in Azure Databricks|
|Change data feed||4||1||Use Delta Lake change data feed on Azure Databricks|
|Generated columns||4||1||Use Delta Lake generated columns|
|Column mapping||5||2||Rename and drop columns with Delta Lake column mapping|
|Identity columns||6||2||Use identity columns in Delta Lake|
|Table features read||7||1||What are table features?|
|Table features write||7||3||What are table features?|
|Deletion vectors||7||3||What are deletion vectors?|
|Liquid clustering||7||3||Use liquid clustering for Delta tables|
|UniForm||7||2||Universal Format (UniForm) for Iceberg compatibility with Delta tables|
You can choose to manually update a table to a newer protocol version. Databricks recommends using the lowest protocol versions that support the Delta Lake features required for your table. Upgrading the writer protocol might cause less disruption than upgrading the reader protocol since systems and workloads using older Databricks Runtime versions can still read from tables, even if they do not support the updated writer protocol.
Protocol version upgrades are irreversible, and upgrading the protocol version might break the existing Delta Lake table readers, writers, or both. Databricks recommends you upgrade specific tables only when needed, such as to opt-in to new features in Delta Lake. You should also check to make sure that all of your current and future production tools support Delta Lake tables with the new protocol version.
To upgrade a table to a newer protocol version, use the
-- Upgrades the reader protocol version to 1 and the writer protocol version to 3. ALTER TABLE <table-identifier> SET TBLPROPERTIES('delta.minReaderVersion' = '1', 'delta.minWriterVersion' = '3')
from delta.tables import DeltaTable delta = DeltaTable.forPath(spark, "path_to_table") # or DeltaTable.forName delta.upgradeTableProtocol(1, 3) # upgrades to readerVersion=1, writerVersion=3
import io.delta.tables.DeltaTable val delta = DeltaTable.forPath(spark, "path_to_table") // or DeltaTable.forName delta.upgradeTableProtocol(1, 3) // Upgrades to readerVersion=1, writerVersion=3.