Drop a Delta Lake table feature and downgrade table protocol

In Databricks Runtime 16.4 LTS and above, DROP FEATURE removes a Delta Lake table feature and downgrades the table protocol to the lowest version that supports the remaining features. Use this only to restore compatibility with earlier Databricks Runtime versions, OpenSharing, or external Delta Lake clients.

Not all features can be dropped. See Supported table features.

Note

Legacy support for DROP FEATURE is available starting in Databricks Runtime 14.3 LTS. Databricks recommends using Databricks Runtime 16.3 and above for all DROP FEATURE commands, which replaces the legacy behavior. For documentation of the legacy functionality, see Drop Delta table features (legacy).

Drop a Delta Lake feature

Important

All DROP FEATURE operations conflict with all concurrent writes.

Streaming reads fail when they encounter a commit that changes table metadata. You must restart affected streams. For recommended methods, see Production considerations for Structured Streaming.

You must use Databricks Runtime 16.3 or above and have MODIFY privileges on the target Delta Lake table. You can only drop one table feature with each DROP FEATURE command.

To drop a table feature, use the following syntax:

ALTER TABLE <table-name> DROP FEATURE <feature-name>

See ALTER TABLE for more details.

Behavior

When you drop a table feature, Delta Lake atomically commits the following changes to the table:

  • Turn off table properties that use the table feature.
  • Rewrite underlying data files as necessary to remove all traces of the table feature from the current table version.
  • Create a set of protected checkpoints that allow reader clients to read the table history correctly.
  • Add the writer table feature checkpointProtection to the table protocol.
  • Downgrade the table protocol to the lowest reader and writer versions that support all remaining table features. See Lowest possible protocol.

The checkpointProtection table feature

When you drop a feature, Delta Lake rewrites data and metadata in the table's history as protected checkpoints to respect the protocol downgrade.

After the downgrade, the table doesn't require clients to support the dropped feature. The protected checkpoints and the checkpointProtection feature accomplish the following:

  • Reader clients that support the dropped table feature can read all available table history versions.
  • Reader clients that don't support the dropped feature can read table history versions from the protocol downgrade version forward.
  • Writer clients don't modify checkpoints created before the protocol downgrade.
  • checkpointProtection marks protocol downgrade checkpoints as protected, and table maintenance operations respect those markers.

While you can only drop one table feature with each DROP FEATURE command, a table can have multiple protected checkpoints and dropped features in its table history.

All Databricks Runtime versions support the checkpointProtection table feature, meaning this table feature doesn't block reads or writes on Azure Databricks.

checkpointProtection doesn't block read-only access from OSS Delta Lake clients. To fully downgrade the table and remove the checkpointProtection table feature, you must use TRUNCATE HISTORY. Databricks recommends only using this pattern if you need to write to tables with external Delta clients that don't support checkpointProtection. See Fully downgrade table protocols for legacy clients.

Supported table features

You can drop the following Delta Lake table features:

You cannot drop other Delta Lake table features.

Important

Databricks recommends that you always test dependent workloads and systems for compatibility with new features before enabling features that upgrades reader or writer protocols for production data.

Consider the following:

  • Dropping column mapping from a table does not remove the random prefixes used in directory names for partitioned tables. See Do Delta Lake and Parquet share partitioning strategies?.
  • Some Delta Lake features enables multiple table features that depend on each other, which might block you from dropping dependent features. You can't roll back features that can't be dropped.

Fully downgrade table protocols for legacy clients

If you use external Delta Lake clients with writes that don't support the checkpointProtection table feature, you must use TRUNCATE HISTORY to fully downgrade the table protocol and remove all traces of the removed table features. Running TRUNCATE HISTORY removes all table history older than 24 hours.

Databricks recommends testing the default behavior for DROP FEATURE before running TRUNCATE HISTORY.

The full downgrade requires two steps, at least 24 hours apart.

Step 1: Prepare to drop a table feature

Run the ALTER TABLE <table-name> DROP FEATURE <feature-name> TRUNCATE HISTORY command. The command applies the following:

  • Updates table properties to turn off the feature.
  • Resets associated table properties for the feature to their defaults.
  • Rewrites data and metadata files as needed to reflect the updated properties.
  • Returns a message requiring you to wait at least 24 hours before running it again to complete the protocol downgrade.

After you turn off a feature, you can continue writing to the target table before completing the protocol downgrade, but you can't use the table feature you are removing.

Note

If you leave the table in this state, operations against the table don't use the table feature, but the protocol still supports the table feature. Until you complete the final downgrade step, the table isn't readable by Delta clients that don't support the table feature.

Step 2: Downgrade the protocol and drop a table feature

After at least 24 hours, run the ALTER TABLE <table-name> DROP FEATURE <feature-name> TRUNCATE HISTORY command again. The command applies the following:

  • Truncates table history to the retention threshold.
  • Confirms no remaining transactions use the feature.
  • Drops the feature and downgrades the table protocol to the lowest version that supports the remaining features.

Important

Running ALTER TABLE <table-name> DROP FEATURE <feature-name> TRUNCATE HISTORY removes all transaction log data older than 24 hours. After using this command to downgrade the table protocol, you can't access table history or time travel.

See Delta Lake feature compatibility and protocols.

  • Last updated on