Work with table history

Each operation that modifies a table creates a new table version. Use history information to audit operations, rollback a table, or query a table at a specific point in time using time travel.

Note

Databricks doesn't recommend using table history as a long-term backup solution for data archival. Use only the past 7 days for time travel operations unless you have set both data and log retention configurations to a larger value.

Retrieve table history

Retrieve information including the operations, user, and timestamp for each write to a table by running the history command. The operations are returned in reverse chronological order.

Table history retention is determined by the table setting logRetentionDuration, which is 30 days by default.

Note

Time travel and table history are controlled by different retention thresholds. See What is time travel?.

DESCRIBE HISTORY table_name       -- get the full history of the table

DESCRIBE HISTORY table_name LIMIT 1  -- get the last operation only

For Spark SQL syntax details, see DESCRIBE HISTORY.

See the Delta Lake API documentation for Scala/Java/Python syntax details.

Catalog Explorer provides a visual view of this detailed table information and history. In addition to the table schema and sample data, you can click the History tab to see the table history that displays with DESCRIBE HISTORY.

History schema

The output of the history operation has the following columns.

Column	Type	Description
version	`long`	Table version generated by the operation.
timestamp	`timestamp`	When this version was committed.
userId	`string`	ID of the user that ran the operation.
userName	`string`	Name of the user that ran the operation.
operation	`string`	Name of the operation.
operationParameters	`map`	Parameters of the operation (for example, predicates.)
job	`struct`	Details of the Lakeflow job that ran the operation. Populates only for commits written from a Lakeflow job. Otherwise, `null`.
notebook	`struct`	Details of the Databricks notebook from which the operation was run. Populates only for commits written from a Databricks notebook. Otherwise, `null`.
clusterId	`string`	ID of the cluster on which the operation ran.
readVersion	`long`	Version of the table that was read to perform the write operation.
isolationLevel	`string`	Isolation level used for this operation.
isBlindAppend	`boolean`	Whether this operation appended data.
operationMetrics	`map`	Metrics of the operation (for example, number of rows and files modified.)
userMetadata	`string`	User-defined commit metadata if it was specified.

+-------+-------------------+------+--------+---------+--------------------+----+--------+---------+-----------+-----------------+-------------+--------------------+
|version|          timestamp|userId|userName|operation| operationParameters| job|notebook|clusterId|readVersion|   isolationLevel|isBlindAppend|    operationMetrics|
+-------+-------------------+------+--------+---------+--------------------+----+--------+---------+-----------+-----------------+-------------+--------------------+
|      5|2019-07-29 14:07:47|   ###|     ###|   DELETE|[predicate -> ["(...|null|     ###|      ###|          4|WriteSerializable|        false|[numTotalRows -> ...|
|      4|2019-07-29 14:07:41|   ###|     ###|   UPDATE|[predicate -> (id...|null|     ###|      ###|          3|WriteSerializable|        false|[numTotalRows -> ...|
|      3|2019-07-29 14:07:29|   ###|     ###|   DELETE|[predicate -> ["(...|null|     ###|      ###|          2|WriteSerializable|        false|[numTotalRows -> ...|
|      2|2019-07-29 14:06:56|   ###|     ###|   UPDATE|[predicate -> (id...|null|     ###|      ###|          1|WriteSerializable|        false|[numTotalRows -> ...|
|      1|2019-07-29 14:04:31|   ###|     ###|   DELETE|[predicate -> ["(...|null|     ###|      ###|          0|WriteSerializable|        false|[numTotalRows -> ...|
|      0|2019-07-29 14:01:40|   ###|     ###|    WRITE|[mode -> ErrorIfE...|null|     ###|      ###|       null|WriteSerializable|         true|[numFiles -> 2, n...|
+-------+-------------------+------+--------+---------+--------------------+----+--------+---------+-----------+-----------------+-------------+--------------------+

Note

A few of the other columns are not available if you write into a table using the following methods:
Columns added in the future will always be added after the last column.

Understanding `partitionBy` in operation parameters

The partitionBy field is only meaningful for CREATE and OVERWRITE operations that define or change a table's partition schema.

For append operations to existing tables (APPEND, INSERT, UPDATE, DELETE, MERGE), this field might show an empty array [] or partition columns depending on the write method used (.save() vs .saveAsTable()). This inconsistency is expected behavior and should not be used to validate writes.

Important

Do not rely on partitionBy in history to validate append operations. The value varies based on implementation details but does not affect how data is written to partitions.

Example

Consider a table partitioned by the date column:

# Initial table creation - partitionBy is populated
df.write.format("delta") \
  .partitionBy("date") \
  .saveAsTable("sales_data")

The CREATE operation in history shows:

operationParameters: {
  "mode": "ErrorIfExists",
  "partitionBy": "[\"date\"]"
}

When you append data to this table:

# Subsequent append - partitionBy shows empty
new_df.write.format("delta") \
  .mode("append") \
  .saveAsTable("sales_data")

The APPEND operation shows:

operationParameters: {
  "mode": "Append",
  "partitionBy": "[]"
}

The empty partitionBy value is expected. The data is still written to the correct partitions based on the table's existing partition schema. Note that .save() to a path may show partition columns in this field, but this difference is an implementation detail and does not affect write behavior.

Operation metrics

The history operation returns a collection of operations metrics in the operationMetrics column map.

The following tables list the map key definitions by operation.