Drop a Delta Lake table feature and downgrade table protocol

This article describes how to drop Delta Lake table features and downgrade protocol versions.

This functionality is available in Databricks Runtime 16.3 and above. Not all Delta table features can be dropped. See What Delta table features can be dropped?.

You should only use DROP FEATURE to support compatibility with earlier Databricks Runtime versions, Delta Sharing, or external Delta Lake reader or writer clients.

Note

Legacy support for DROP FEATURE is available starting in Databricks Runtime 14.3 LTS. Databricks recommends using Databricks Runtime 16.3 and above for all DROP FEATURE commands, which replaces the legacy behavior. For documentation of the legacy functionality, see Drop Delta table features (legacy).

Drop a Delta Lake feature

To drop a table feature, use the following syntax:

ALTER TABLE <table-name> DROP FEATURE <feature-name>

You must use Databricks Runtime 16.3 or above and have MODIFY privileges on the target Delta table. You can only drop one table feature with each DROP FEATURE command.

See ALTER TABLE for more details.

Important

All DROP FEATURE operations conflict with all concurrent writes.

Streaming reads fail when they encounter a commit that changes table metadata. If you want the stream to continue you must restart it. For recommended methods, see Production considerations for Structured Streaming.

What happens when a table feature is dropped?

When you drop a table feature, Delta Lake atomically commits changes to the table to accomplish the following:

• Disable table properties that use the table feature.
• Remove all traces of the table feature from the data files backing the table in the current version.
• Create a set of protected checkpoints that allow reader clients to interpret table history correctly.
• Add the writer table feature checkpointProtection to the table protocol.
• Downgrade the table protocol to the lowest reader and writer versions that support all remaining table features. See Lowest possible protocol.

What is the checkpointProtection table feature?

When you drop a feature, Delta Lake rewrites data and metadata in the table's history as protected checkpoints to respect the protocol downgrade. After the downgrade, the table should always be readable by more reader clients. This is because the protocol for the table now reflects that support for the dropped feature is no longer required to read the table. The protected checkpoints and the checkpointProtection feature accomplish the following:

• Reader clients that understand the dropped table feature can access all available table history.
• Reader clients that do not support the dropped table feature only need to read the table history starting from the protocol downgrade version.
• Writer clients do not rewrite checkpoints prior to the protocol downgrade.
• Table maintenance operations respect requirements set by checkpointProtection, which mark protocol downgrade checkpoints as protected.

While you can only drop one table feature with each DROP FEATURE command, a table can have multiple protected checkpoints and dropped features in its table history.

All Databricks Runtime versions support the checkpointProtection table feature, meaning this table feature does not block reads or writes on Azure Databricks.

The table feature checkpointProtection should not block read-only access from OSS Delta Lake clients. To fully downgrade the table and remove the checkpointProtection table feature, you must use TRUNCATE HISTORY. Databricks recommends only using this pattern if you need to write to tables with external Delta clients that do not support checkpointProtection. See Fully downgrade table protocols for legacy clients.

What Delta table features can be dropped?

You can drop the following Delta table features:

• checkConstraints. See Constraints on Azure Databricks.
• collations-preview. See Collation support for Delta Lake.
• columnMapping. See Rename and drop columns with Delta Lake column mapping.
• deletionVectors. See What are deletion vectors?.
• typeWidening-preview. See Type widening.
• v2Checkpoint. See Compatibility for tables with liquid clustering.
• checkpointProtection. See What is the checkpointProtection table feature?.

You cannot drop other Delta table features.

Important

Some Delta Lake functionality enables multiple table features. Some table features depend on other table features, and might block dropping dependent table features. Because some table features cannot be dropped, this means that enablement for some Delta Lake features cannot be rolled back.

Databricks recommends always testing dependent workloads and systems for compatibility with new functionality before enabling functionality that upgrades reader or writer protocols for production data.

Fully downgrade table protocols for legacy clients

If integrations with external Delta Lake clients require writes that don't support the checkpointProtection table feature, you must use TRUNCATE HISTORY to fully remove all traces of the disabled table features and fully downgrade the table protocol.

Databricks recommends testing the default behavior for DROP FEATURE before proceeding with TRUNCATE HISTORY. Running TRUNCATE HISTORY removes all table history greater than 24 hours.

Full table downgrade occurs in two steps that must occur at least 24 hours apart.

Step 1: Prepare to drop a table feature

During the first stage, the user prepares to drop the table feature. The following describes what happens during this stage:

1. You run the ALTER TABLE <table-name> DROP FEATURE <feature-name> TRUNCATE HISTORY command.
2. Table properties that specifically enable a table feature have values set to disable the feature.
3. Table properties that control behaviors associated with the dropped feature have options set to default values before the feature was introduced.
4. As necessary, data and metadata files are rewritten respecting the updated table properties.
5. The command finishes running and returns an error message informing the user they must wait 24 hours to proceed with feature removal.

After first disabling a feature, you can continue writing to the target table before completing the protocol downgrade, but you cannot use the table feature you are removing.

Note

If you leave the table in this state, operations against the table do not use the table feature, but the protocol still supports the table feature. Until you complete the final downgrade step, the table is not readable by Delta clients that do not understand the table feature.

Step 2: Downgrade the protocol and drop a table feature

To fully remove all transaction history associated with the feature and downgrade the protocol:

1. After at least 24 hours have passed, you run the ALTER TABLE <table-name> DROP FEATURE <feature-name> TRUNCATE HISTORY command.
2. The client confirms that no transactions in the specified retention threshold use the table feature, then truncates the table history to that threshold.
3. The table feature is dropped during protocol downgrade.
4. If the table features that are present in the table can be represented by a lower protocol version, the minReaderVersion and minWriterVersion for the table are downgraded to the lowest version that supports the remaining features in use by the Delta table.

Important

Running ALTER TABLE <table-name> DROP FEATURE <feature-name> TRUNCATE HISTORY removes all transaction log data older than 24 hours. After using this command to downgrade the table protocol, you do not have access to table history or time travel.

See Delta Lake feature compatibility and protocols.