Deploy the ETL code from Dev Synapse Analytics workspace to Another Workspace (different Resource group and Different subscription)

Question

Hi all,

I had an issue earlier and I had a great response and it has been resolved now.

I need to know what's the best approach to:

1. Deploy all ETL code (pipelines/ copy , data flows) produced on Synapse and stored on DevOps on different branches to another workspace within the same subscription/tenant but different resource group. Service connection have changed, data lake storage have changed. I need to be able to deploy the code on that workspace and rerun the whole thing.
2. Once step is done I need repeat the same process using DevOps to deploy the same code to another workspace which is the client's new Subscription/New tenant and everything is new (data lake, linked service) . Once we deploy to client's env we need to be able to apply the same changes (incremental) from our Dev workspace to Clients Synapse workspace.

We have access to client's subscription and have service principal which have assigned the contributor role. All resource are created on clients env

I will start with this: https://learn.microsoft.com/en-us/azure/synapse-analytics/cicd/continuous-integration-delivery

Please can you advise what's the best approach to do this for both scenarios.

Will appreciate any help.

Thanks

Answer 1

@Xhevahir Mehalla - Thanks for the question and using MS Q&A platform.

Regarding your question, the best approach to deploy ETL code from one Synapse workspace to another Synapse workspace in the same subscription but different resource group is to use the Synapse Workspace Deployment tool in Azure DevOps or GitHub.

Here are the high-level steps to deploy ETL code from one Synapse workspace to another Synapse workspace in the same subscription but different resource group:

1. Create an Azure Resource Manager (ARM) template to create or update workspace resources (pools and workspace) in the target Synapse workspace.
2. Migrate artifacts like SQL scripts and notebooks, Spark job definitions, pipelines, datasets, and other artifacts by using the Synapse Workspace Deployment tool in Azure DevOps or GitHub.
3. Update the service connection and data lake storage in the target Synapse workspace.

Regarding your second scenario, to deploy the same code to another workspace in a different subscription or tenant, you can follow the same steps as above. However, you will need to create a new service connection and data lake storage in the client's environment. You can use the Azure DevOps release pipeline and GitHub Actions to automate the deployment of the Synapse workspace to the client's environment.

For more details, refer to BCDR for Azure Data Factory and Azure Synapse Analytics pipelines and Transfer an entire Azure Synapse Analytics workspace to another subscription

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.