Sync the tables from one azure databricks workspace to other databricks workspace or to adls gen2

Ashwini Gaikwad 65

Hello Team,

We have two UC enabled databricks workspace. And we have to sync tables created in one Azure databricks workspace to other databricks workspace using PAT/any other reliable way or to adls gen2 account. Request you to let me know is there a way to implement it via Azure Datafactory or any other way, how it can be achieved and is there any relevant article for it to follow.

Regards,

Ashwini G

Ashwini Gaikwad 65 Reputation points

2024-04-12T07:36:32.95+00:00

@BhargavaGunnam-MSFT thank you for your response. Point 4 explained by you would be the ideal way for us going forward but how can I implement it. Do I need to used delta lake connector? How can I sync tables from one UC workspace to adls gen2? Can you suggest step by step way to implement via copy activity or share the reference article?
BhargavaGunnam-MSFT 26,306 Reputation points Microsoft Employee

2024-04-12T20:05:49.68+00:00

Hello Ashwini Gaikwad,

You can use the Azure Databricks Delta Lake connector to copy data from one Azure Databricks workspace to another Databricks workspace.

Copy activity supports Azure Databricks Delta Lake connector to copy data from any supported source data store to Azure Databricks delta lake table, and from delta lake table to any supported sink data store. It leverages your Databricks cluster to perform the data movement.

In your scenario, the source and destination are both Azure Databricks Delta Lake connectors.

You can follow the below reference document:

https://learn.microsoft.com/en-us/azure/data-factory/connector-azure-databricks-delta-lake?tabs=data-factory

I hope this answers your question. Please let me know if you have any further questions.
BhargavaGunnam-MSFT 26,306 Reputation points Microsoft Employee

2024-04-17T17:54:35.2733333+00:00

Hello Ashwini Gaikwad,

I am checking to see if you have any further questions here.
Ashwini Gaikwad 65 Reputation points

2024-04-22T16:57:36.8133333+00:00

@BhargavaGunnam-MSFT thank you for your response. I will implement the solution with Delta lake connector and will get back to you in case of any queries

1 answer

BhargavaGunnam-MSFT 26,306 Reputation points Microsoft Employee

2024-04-11T21:16:40.9833333+00:00
Hello Ashwini Gaikwad,

I see the below different methods that Synchronizing tables between two Azure Databricks workspaces

creating a Hive external metastore that multiple Databricks workspaces can share. Each workspace can register and use this commonly shared metastore.

Please check the below document. Multiple scenario use cases are explained.

Reference document: https://techcommunity.microsoft.com/t5/fasttrack-for-azure/sharing-metadata-across-different-databricks-workspaces-using/ba-p/3679757

If you have tables in one workspace that you want to move to another, consider using external tables as explained in the below databricks community forum.

https://community.databricks.com/t5/data-engineering/migrate-tables-from-one-azure-databricks-workspace-to-another/td-p/23358

You can also use the DBSync project. This is an object synchronization tool that backs up, restores, and syncs Databricks workspaces.

https://github.com/databrickslabs/databricks-sync

You can use ADF copy activity to copy the data from one Azure databricks to another databricks workspace.

I hope this helps. Please let me know if you have any further questions.
Please sign in to rate this answer.
Ashwini Gaikwad 65 Reputation points

2024-04-25T10:38:55.3233333+00:00

Hello @BhargavaGunnam-MSFT I need to read all tables in one particular database in Databricks workspace which is not UC enabled. How can I achieve that via ADF and using delta lake connector? I don't see any relevant documents for the same. Can you please help me in this case?

BhargavaGunnam-MSFT 26,306 Reputation points Microsoft Employee

2024-04-26T01:54:54.37+00:00

Please follow the below document:

https://learn.microsoft.com/en-us/azure/data-factory/connector-azure-databricks-delta-lake?tabs=data-factory#access-token

Ashwini Gaikwad 65 Reputation points

2024-04-26T08:28:43.58+00:00

Hello @BhargavaGunnam-MSFT Thank you for your response. I have gone through the article which you have shared but it doesn't answer how to read all tables in one database located in hive metastore. Let say in my databricks hive metastore I have a database abc and there are multiple tables inside it for e.g crm, odbc, isnta. So how can I read all this tables or query those using delta lake connector and copy it to adls gen2 account in delta parquet format? What could be the most generic solution to implement this?

Ashwini Gaikwad 65 Reputation points

2024-04-26T08:33:27.6966667+00:00

@BhargavaGunnam-MSFT thank you so much for your response.

I have referred the article shared by you but it doesn't answer how should I query all the tables from one database stored in databricks hive metastore. For e.g. I have a database abc and tables residing under it are crm, cro, odbc and so on. So how should I query or read those delta tables from azure delta lake connector and copy it into adls gen2 account using parquet format?

What could be the more generic solution for this?

BhargavaGunnam-MSFT 26,306 Reputation points Microsoft Employee

2024-04-26T18:25:52.01+00:00

Thanks for the additonal details.

To read all tables from your Databricks Hive metastore database 'abc' and copy them to an ADLS Gen2 account in Delta Parquet format,- You can use spark SQL with in databricks to query hive metastore and read the tables. and then write to gen2 in delta parquet format using delta lake connector.

Please see the below reference stackoverflow threads

https://stackoverflow.com/questions/63662988/how-to-show-all-tables-in-all-databases-in-databricks

https://stackoverflow.com/questions/56006982/databricks-failing-to-write-from-a-dataframe-to-a-delta-location
Sign in to comment