Set Delta table properties

Applies to: SaaS connectors Database connectors Query-based connectors

You can set Delta Lake table properties on the destination streaming tables that a managed ingestion pipeline creates. Because destination tables are streaming tables managed by the pipeline, you set table properties through the pipeline's ingestion_definition rather than by running ALTER TABLE ... SET TBLPROPERTIES directly. Specifying table_properties in the pipeline spec has the same effect as the ALTER TABLE ... SET TBLPROPERTIES command.

You can set table properties at two levels:

Level	Behavior
Pipeline	Set table_properties in the pipeline-level table_configuration to apply the properties to all tables and schemas in the pipeline.
Table or schema	Set table_properties in the table_configuration of an individual table or schema to override the pipeline-level configuration for that object.

Warning

Before you enable a table property, verify the reader and writer version requirements for that feature. For property-specific requirements, see Table properties reference and the documentation for the feature you're enabling, such as Type widening.

Set table properties using the API

You can set table properties when you create or edit managed ingestion pipelines using Declarative Automation Bundles, notebooks, or the Databricks CLI.

The following example enables type widening at the pipeline level, and then overrides it to false for the users_1 destination table. The calendar_test table inherits the pipeline-level setting.

Declarative Automation Bundles

resources:
  pipelines:
    pipeline_cdc:
      name: <pipeline-name>
      catalog: <destination-catalog>
      schema: <destination-schema>
      ingestion_definition:
        ingestion_gateway_id: <gateway-pipeline-id>
        table_configuration:
          table_properties:
            delta.enableTypeWidening: 'true'
            delta.enableDeletionVectors: 'true'
        objects:
          - table:
              source_schema: <source-schema>
              source_table: users_1
              destination_catalog: <destination-catalog>
              destination_schema: <destination-schema>
              table_configuration:
                table_properties:
                  delta.enableTypeWidening: 'false'
          - table:
              source_schema: <source-schema>
              source_table: calendar_test
              destination_catalog: <destination-catalog>
              destination_schema: <destination-schema>

Databricks notebook

pipeline_spec = """
{
  "ingestion_definition": {
    "ingestion_gateway_id": "<gateway-pipeline-id>",
    "table_configuration": {
      "table_properties": {
        "delta.enableTypeWidening": "true",
        "delta.enableDeletionVectors": "true"
      }
    },
    "objects": [
      {
        "table": {
          "source_schema": "<source-schema>",
          "source_table": "users_1",
          "destination_catalog": "<destination-catalog>",
          "destination_schema": "<destination-schema>",
          "table_configuration": {
            "table_properties": {
              "delta.enableTypeWidening": "false"
            }
          }
        }
      },
      {
        "table": {
          "source_schema": "<source-schema>",
          "source_table": "calendar_test",
          "destination_catalog": "<destination-catalog>",
          "destination_schema": "<destination-schema>"
        }
      }
    ]
  }
}
"""

Databricks CLI

{
  "ingestion_definition": {
    "ingestion_gateway_id": "<gateway-pipeline-id>",
    "table_configuration": {
      "table_properties": {
        "delta.enableTypeWidening": "true",
        "delta.enableDeletionVectors": "true"
      }
    },
    "objects": [
      {
        "table": {
          "source_schema": "<source-schema>",
          "source_table": "users_1",
          "destination_catalog": "<destination-catalog>",
          "destination_schema": "<destination-schema>",
          "table_configuration": {
            "table_properties": {
              "delta.enableTypeWidening": "false"
            }
          }
        }
      },
      {
        "table": {
          "source_schema": "<source-schema>",
          "source_table": "calendar_test",
          "destination_catalog": "<destination-catalog>",
          "destination_schema": "<destination-schema>"
        }
      }
    ]
  }
}

You can specify multiple table properties in a single table_properties block.

Inheritance and overrides

Table properties follow an inheritance model, where settings applied to an individual table or schema take precedence over pipeline-level defaults.

• Table properties set at the pipeline level are inherited by all tables and schemas in the pipeline.
• A table_configuration set at the table or schema level overrides the pipeline-level configuration. When you override at the table or schema level, other pipeline-level table configurations are not inherited. To keep a pipeline-level configuration, repeat it in the override.

Limitations

The following limitations apply when setting table properties on managed ingestion pipelines.

• Setting table properties for managed ingestion pipelines is only supported for database connectors that use change data capture (CDC). SaaS connectors and query-based connectors are not supported.
• Removing a table property from the pipeline spec does not automatically revert the underlying table feature. For example, if you enable delta.enableTypeWidening and later find that your reader Databricks Runtime version doesn't support type widening, removing the property from the spec doesn't undo the change. In that case, you must perform a full refresh of the affected tables. Before you change a table property, confirm whether reverting the property resolves the issue or whether a full refresh is required. See Fully refresh target tables.

Additional resources

• Table properties reference
• Type widening
• Enable liquid clustering