Use Apache Iceberg v3 features

Apache Iceberg v3 enhances query performance and introduces new capabilities for managed tables using Iceberg or Delta Lake with UniForm in Unity Catalog.

The key features of Iceberg v3 are:

Requirements

To use Iceberg v3 features, you must meet the following requirements:

  • A workspace with Unity Catalog enabled.
  • Databricks Runtime 18.0 or above to read and write to managed tables with Iceberg v3.
  • Databricks Runtime 18.2 or above to use geospatial types with Iceberg v3.

Create a new table with Iceberg v3

Create new tables with Iceberg v3 enabled for both managed Delta Lake tables with UniForm and managed Iceberg tables.

Delta Lake

To create a new managed table using Delta Lake with UniForm and Iceberg v3 enabled, use the following SQL command:

CREATE OR REPLACE TABLE main.schema.table (c1 INT) TBLPROPERTIES(
  'delta.universalFormat.enabledFormats' = 'iceberg',
  'delta.enableIcebergCompatV3' = 'true'
);

For more information about UniForm, see Read Delta Lake tables with Iceberg clients.

Iceberg

To create a new managed table using Iceberg v3, use the following SQL command:

CREATE OR REPLACE TABLE main.schema.table (c1 INT)
USING iceberg
TBLPROPERTIES ('format-version' = 3);

For more information about Iceberg tables, see What is Apache Iceberg in Azure Databricks?.

Upgrade an existing table to Iceberg v3

You can upgrade an existing table to Iceberg v3 by:

  1. Enabling any v3 feature on a table.
  2. Setting the Iceberg format version on a table to 3 (shown below).

Important

Tables can be downgraded from v3 to v2 by restoring the table to a table version before the upgrade to v3 using RESTORE. See Downgrade a table to a previous version.

Delta Lake

To upgrade a managed table using Delta Lake with UniForm to v3, use the following command:

ALTER TABLE catalog.schema.table SET TBLPROPERTIES(
  'delta.enableIcebergCompatV3' = 'true',
  'delta.enableIcebergCompatV2' = 'false'
);

Iceberg

To upgrade a managed table using Iceberg to v3, use the following command:

ALTER TABLE catalog.schema.table SET TBLPROPERTIES (
  'format-version' = 3
);

Enable deletion vectors

Deletion vectors optimize row-level data modification operations and are enabled by default on all new Iceberg v3 tables. See Deletion vectors in Databricks.

Note

Enabling deletion vectors on an existing Iceberg table upgrades the Iceberg format version to 3.

Delta Lake

To create a new managed table using Delta Lake with UniForm, Iceberg v3, and deletion vectors enabled, set the following table properties:

CREATE TABLE catalog.schema.table (c1 INT) TBLPROPERTIES(
  'delta.enableDeletionVectors' = 'true',
  'delta.enableIcebergCompatV3' = 'true',
  'delta.universalFormat.enabledFormats' = 'iceberg'
);

Iceberg

To create a new managed table using Iceberg v3 with deletion vectors enabled, set the iceberg.enableDeletionVectors table property:

CREATE TABLE catalog.schema.table (c1 INT)
USING ICEBERG TBLPROPERTIES (
  'iceberg.enableDeletionVectors' = 'true'
);

Use the VARIANT data type

The VARIANT data type allows you to store and query semi-structured data.

Note

Using VARIANT in an existing Iceberg table upgrades the Iceberg format version to 3.

Delta Lake

To create a new managed table using Delta Lake with UniForm and a VARIANT column:

CREATE TABLE catalog.schema.deltaTable (col VARIANT) TBLPROPERTIES(
  'delta.enableIcebergCompatV3' = 'true',
  'delta.universalFormat.enabledFormats' = 'iceberg'
);

Iceberg

To create a new managed table using Iceberg v3 with a VARIANT column:

CREATE TABLE catalog.schema.icebergTable (col VARIANT) USING iceberg;

To add a VARIANT column to an existing table, use the ALTER TABLE command:

ALTER TABLE catalog.schema.table ADD COLUMN variant_col VARIANT;

Downgrade a table to a previous Apache Iceberg version

If you need to revert a table to a state before it was upgraded to Iceberg v3, you can use the RESTORE command.

set spark.databricks.delta.restore.protocolDowngradeAllowed = true;
RESTORE TABLE catalog.schema.table TO VERSION AS OF 1;
set spark.databricks.delta.restore.protocolDowngradeAllowed = false;

Limitations

Azure Databricks supports version 3 of the Iceberg specification, with the following exceptions:

  • Defaults, including write defaults and initial defaults, aren't supported.
  • The following data types aren't supported:
    • Unknown type
    • Nanosecond precision timestamp.
  • Multi-argument transforms aren't supported.
  • Last updated on