Azure Data Factory V2 Arm template to Synpase Data Integration without Git

Dondapati, Navin 281 Reputation points
2021-05-21T03:17:44.68+00:00

Hi Guys,

We have Azure data factory as separate resources within same resource group, now we exported out the arm template from ADF. And we recently spin up Azure synapse component on azure; We do see Data integration component which is similar to ADF, But we could not find a way to deploy our old pipelines to Azure Synapse.

How can i deploy my arm template to synapse and make sure my pipeline works fine?

Regards,
Navin

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,117 questions
0 comments No comments
{count} votes

Accepted answer
  1. PRADEEPCHEEKATLA-MSFT 73,731 Reputation points Microsoft Employee
    2021-05-21T11:19:07.147+00:00

    Hello @Anonymous ,

    Thanks for the question and using MS Q&A platform.

    Currently, there is no direct method to migrate ADF v2 templates to Azure Synapse Analytics.

    Alternate Solution:

    You can use Azure PowerShell to upload the downloaded JSON supported files to Azure Synapse Analytics and use it.

    Here in the below example I'm trying to move ADF pipeline from (ADF:chepra) to Azure Synapse Analytics Workspace (Synapse Analytics: cheprasynapse).

    After upload the JSON files to Azure Synapse Analytics workspace, make sure to modify the linked services configuration. Else, you get this error message Cloud stored credentials cannot be found with retry, please check whether related resource had been deleted before.

    Step1: Once you download the supported JSON files from the ADF.

    97411-image.png

    Step2: Log in to PowerShell (Make sure to use other subscription where you want to upload the JSON files)

    Run the following command, and enter the same Azure user name and password that you use to sign in to the Azure portal:  
      
    Connect-AzAccount  
      
    Run the following command to view all the subscriptions for this account:  
      
    Get-AzSubscription  
      
    If you see multiple subscriptions associated with your account, run the following command to select the subscription that you want to work with. Replace SubscriptionId with the ID/Name of your Azure subscription:  
      
    Select-AzSubscription -SubscriptionName "<SubscriptionName>"  
    

    98469-image.png

    Step3: To upload downloaded JSON support files from ADF v2 to Azure Synapse Analytics workspace .

        To Create a linked service:  
          
        Set-AzSynapseLinkedService -WorkspaceName ContosoWorkspace -Name ContosoLinkedService -DefinitionFile "C:\\samples\\LinkedService.json"  
          
        To create a dataset:   
          
       Set-AzSynapseDataset -WorkspaceName ContosoWorkspace -Name ContosoDataset -DefinitionFile "C:\\samples\\Dataset.json"  
          
        To create a pipeline:  
          
        Set-AzSynapsePipeline -WorkspaceName ContosoWorkspace -Name ContosoPipeline -DefinitionFile "C:\pipeline.json"  
    

    98586-image.png

    Now we had successfully uploaded the downloaded JSON files ADF to Azure Synapse Analytics Workspace.

    98567-image.png

    For more details, refer Azure PowerShell cmdlets for Azure Synapse Analytics..

    Hope this helps. Do let us know if you any further queries.

    ------------

    Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.

    1 person found this answer helpful.

1 additional answer

Sort by: Most helpful
  1. Saikat Dutta 1 Reputation point
    2022-06-13T05:22:33.413+00:00

    This is good for single pipeline. However we have 1300 pipelines in our environment. How can we do this? Or do we need to write parameterized powershell queries? Surely, there should be support for an easier way out.