DeltaTable versioning corrupt

Question

I have a scenario where I have performed the az_copy from one blob to other all the delta tables .
And accessing these new delta tables from the new blob storage from new databricks workspace the version timestamp of the tables is defaulted to the timestamp of the az_copy timestamp.

Note: I have used Databricks migration tool to get the metastore transfered from one workpace to other.
I am using sql command ---describe history db.table_name to get the versioning
I am using the python command spark.read.format('delta').option("timestampAsOf",date_time).load('/mnt/container_name/delta_table)

How can I fix the versioning of of this delta table in the new workspace and new blob storage.

Thank you

Answer 1

Hello @Chandra ,

This may be the reason. The versioning information is not in the DDL, and may not be stored in the old metastore as well. Most probably it is stored in the delta log. When table is created in the new metastore, the table creation timestamp will be newer than the delta log timestamp, this may cause only one version can be seen if you run describe history command.

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators