Sync the tables from one azure databricks workspace to other databricks workspace or to adls gen2

Ashwini Gaikwad 130 Reputation points
2024-04-11T18:50:09.1566667+00:00

Hello Team,

We have two UC enabled databricks workspace. And we have to sync tables created in one Azure databricks workspace to other databricks workspace using PAT/any other reliable way or to adls gen2 account. Request you to let me know is there a way to implement it via Azure Datafactory or any other way, how it can be achieved and is there any relevant article for it to follow.

Regards,

Ashwini G

Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
2,221 questions
{count} votes

1 answer

Sort by: Most helpful
  1. Bhargava-MSFT 31,116 Reputation points Microsoft Employee
    2024-04-11T21:16:40.9833333+00:00

    Hello Ashwini Gaikwad,

    I see the below different methods that Synchronizing tables between two Azure Databricks workspaces

    1. creating a Hive external metastore that multiple Databricks workspaces can share. Each workspace can register and use this commonly shared metastore.

    Please check the below document. Multiple scenario use cases are explained.

    Reference document: https://techcommunity.microsoft.com/t5/fasttrack-for-azure/sharing-metadata-across-different-databricks-workspaces-using/ba-p/3679757

    1. If you have tables in one workspace that you want to move to another, consider using external tables as explained in the below databricks community forum.

    https://community.databricks.com/t5/data-engineering/migrate-tables-from-one-azure-databricks-workspace-to-another/td-p/23358

    1. You can also use the DBSync project. This is an object synchronization tool that backs up, restores, and syncs Databricks workspaces.

    https://github.com/databrickslabs/databricks-sync

    1. You can use ADF copy activity to copy the data from one Azure databricks to another databricks workspace.

    I hope this helps. Please let me know if you have any further questions.


Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.