Azure Data Factory V2 Arm template to Synpase Data Integration without Git

Question

Hi Guys,

We have Azure data factory as separate resources within same resource group, now we exported out the arm template from ADF. And we recently spin up Azure synapse component on azure; We do see Data integration component which is similar to ADF, But we could not find a way to deploy our old pipelines to Azure Synapse.

How can i deploy my arm template to synapse and make sure my pipeline works fine?

Regards,
Navin

Answer 1

Hello @Anonymous ,

Thanks for the question and using MS Q&A platform.

Currently, there is no direct method to migrate ADF v2 templates to Azure Synapse Analytics.

Alternate Solution:

You can use Azure PowerShell to upload the downloaded JSON supported files to Azure Synapse Analytics and use it.

Here in the below example I'm trying to move ADF pipeline from (ADF:chepra) to Azure Synapse Analytics Workspace (Synapse Analytics: cheprasynapse).

After upload the JSON files to Azure Synapse Analytics workspace, make sure to modify the linked services configuration. Else, you get this error message Cloud stored credentials cannot be found with retry, please check whether related resource had been deleted before.

Step1: Once you download the supported JSON files from the ADF.

Step2: Log in to PowerShell (Make sure to use other subscription where you want to upload the JSON files)

Run the following command, and enter the same Azure user name and password that you use to sign in to the Azure portal:  
  
Connect-AzAccount  
  
Run the following command to view all the subscriptions for this account:  
  
Get-AzSubscription  
  
If you see multiple subscriptions associated with your account, run the following command to select the subscription that you want to work with. Replace SubscriptionId with the ID/Name of your Azure subscription:  
  
Select-AzSubscription -SubscriptionName "<SubscriptionName>"  

Step3: To upload downloaded JSON support files from ADF v2 to Azure Synapse Analytics workspace .

    To Create a linked service:  
      
    Set-AzSynapseLinkedService -WorkspaceName ContosoWorkspace -Name ContosoLinkedService -DefinitionFile "C:\\samples\\LinkedService.json"  
      
    To create a dataset:   
      
   Set-AzSynapseDataset -WorkspaceName ContosoWorkspace -Name ContosoDataset -DefinitionFile "C:\\samples\\Dataset.json"  
      
    To create a pipeline:  
      
    Set-AzSynapsePipeline -WorkspaceName ContosoWorkspace -Name ContosoPipeline -DefinitionFile "C:\pipeline.json"  

Now we had successfully uploaded the downloaded JSON files ADF to Azure Synapse Analytics Workspace.

For more details, refer Azure PowerShell cmdlets for Azure Synapse Analytics..

Hope this helps. Do let us know if you any further queries.

------------

Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.

Answer 2

This is good for single pipeline. However we have 1300 pipelines in our environment. How can we do this? Or do we need to write parameterized powershell queries? Surely, there should be support for an easier way out.