Efficient Data Migration of Databricks from One Region to Another

Question

Efficient Data Migration of Databricks from One Region to Another

Deepa M 20

How can data bricks be migrated from US East 2 to US Central efficiently? The migration includes moving the existing dev and prod setup, which consists of workspace jobs, notebooks, scripts, catalog tables, volumes, and Azure Data Factory (ADF) pipelines.

Looking for guidance on the most efficient methods to accomplish this migration.

Smaran Thoomu 35,375 Reputation points Moderator

2025-07-10T14:28:06.7366667+00:00
Hi Deepa M
Migrating an Azure Databricks environment from US East 2 to US Central involves multiple components and requires a combination of export/import, scripting, and reconfiguration. Here's a high-level and efficient approach to help guide your migration:

Databricks Workspaces (Jobs, Notebooks, Libraries, Clusters)

Export notebooks and jobs using:

Databricks CLI

or REST API (/api/2.0/workspace/export, /api/2.0/jobs/list)

Import to target region using CLI or APIs (/import, /jobs/create)

For libraries and cluster configs, export JSON definitions and recreate in the target workspace.

Use the Terraform Databricks provider for infrastructure-as-code (IaC) management across regions.

Catalog Tables and Volumes (Unity Catalog)

Unity Catalog migration:

Use Delta Sharing (or export data to ADLS and re-register in new workspace)

Recreate metastore objects in the target region.

Use SHOW CREATE TABLE or Unity Catalog APIs to recreate schemas and permissions.

Note: Unity Catalog is region-bound, so a new metastore must be created in US Central.

Volumes (Managed Storage)

If using Volumes, data must be copied out to Azure Data Lake Storage (ADLS) or Blob and re-ingested in the new region.

Use COPY INTO or Databricks File System (DBFS) tools to transfer.

Azure Data Factory Pipelines

ADF pipelines are region-specific, so:

Export your existing pipelines (ARM templates or JSON via UI/API).

Re-deploy in the new region via Azure Portal, PowerShell, or CI/CD.

Update linked services to point to the new Databricks workspace and storage paths.

Secrets and Key Vaults

Recreate or reference Azure Key Vaults in the new region.

Use Databricks CLI/API to re-upload secrets if using the Databricks Secret Scope.

I hope this information helps. Please do let us know if you have any further queries.

Kindly consider upvoting the comment if the information provided is helpful. This can assist other community members in resolving similar issues.
Smaran Thoomu 35,375 Reputation points Moderator

2025-07-11T07:11:12.6+00:00

Deepa M Just checking in to see if the below answer helped. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Answer accepted by question author

0 additional answers

Your answer

Smaran Thoomu 35,375 Reputation points Moderator

2025-07-11T07:11:12.6+00:00

Deepa M Just checking in to see if the below answer helped. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Answer 1

Hello Deepa !

Thank you for posting on Microsoft Learn

Your first step should be planning.

Begin by identifying all components that need to be moved :

workspace configurations
cluster definitions
jobs
Unity Catalog metadata (if in use)
volumes (including files stored in DBFS)
external tools like ADF pipelines

and don't forget to review access control mechanisms such as service principals, managed identities, and user roles, as these will need to be reconfigured in the destination region.

You need to create a new Databricks workspace in the target region (you can use the Azure Portal, Terraform, or ARM templates)

This workspace will mirror the source configuration in terms of pricing tier, networking (such as VNet integration or private endpoints), and identity setup. If you are using infrastructure as code, this step can be automated and version-controlled.

Next, migrate workspace artifacts (notebooks, jobs, and clusters) .

Notebooks can be exported and imported using the Databricks CLI or REST APIs. Jobs can also be exported via the REST API or CLI and recreated in the new workspace.

For clusters and pools, use the API to capture their JSON definitions and then recreate them in the new environment.

When it comes to data and catalog tables, the approach depends on whether you use Unity Catalog. If so, you’ll need to recreate catalogs, schemas, and tables in the target region, as metastores are region-specific and not portable.

You can use DDL scripts (SHOW CREATE TABLE) to replicate schema structures, and copy data via COPY INTO, DBFS, or external storage like ADLS Gen2. For volumes or DBFS files, you can export files locally or to shared storage, then re-import into the new workspace.

ADF pipelines can be exported as ARM templates or managed via Azure DevOps. You’ll need to update linked services to point to the new Databricks workspace and modify any hardcoded notebook paths. These updated pipelines can then be redeployed to a new or existing ADF instance in the target region.

Don't forget that the security configurations must also be reapplied. This includes workspace access permissions, Unity Catalog access control lists, and secret scopes (which cannot be exported and must be recreated manually). You can use the Databricks permissions API or manage them via Terraform if your setup is code-driven.