Enrich tables with custom metadata

Databricks recommends always providing comments for tables and columns in tables. You can generate these comments using AI. See Add AI-generated comments to Unity Catalog objects.

Unity Catalog also provides the ability to tag data. See Apply tags to Unity Catalog securable objects.

Log messages for individual commits to tables in a field in the transaction log.

Set user-defined commit metadata

Specify user-defined strings as metadata in commits using the DataFrameWriter option userMetadata or the SparkSession configuration spark.databricks.delta.commitInfo.userMetadata (Delta) or spark.databricks.iceberg.commitInfo.userMetadata (Iceberg). If both have been specified, the option takes preference. This user-defined metadata is readable in the DESCRIBE HISTORY operation. See Work with table history.

SQL

-- For Delta tables
SET spark.databricks.delta.commitInfo.userMetadata=some-comment
INSERT OVERWRITE target_table SELECT * FROM data_source

-- For Iceberg tables
SET spark.databricks.iceberg.commitInfo.userMetadata=some-comment
INSERT OVERWRITE target_table SELECT * FROM data_source

Python

df.write \
  .mode("overwrite") \
  .option("userMetadata", "some-comment") \
  .table("target_table")

Scala

df.write
  .mode("overwrite")
  .option("userMetadata", "some-comment")
  .table("target_table")