how to create arm template for azure synapse pipeline

Question

Greetings!!!

I'm new to azure synapse analytics and I have a few pipelines in my development workspace which I would like to make it deploy to other synapse analytics workspace using an ARM template. I'm not sure how to create an ARM template to make code deploy to any environment seamlessly. I have looked at this document (https://learn.microsoft.com/en-us/azure/synapse-analytics/quickstart-deployment-template-workspaces) but not sure how to include the linked service, dataset, pipeline, integration runtime and shared integration runtime (if any used).

Thank you

Answer 1

Hello @Raj D,

Welcome to the MS Q&A platform.

You can directly deploy the synapse workspace and the Gen2 account using the URL mentioned in the question. But if you want to deploy your pipelines, datasets etc, you will need to edit the template.

You can also use the custom deployment ARM template to edit your changes from scratch.

But below is the limitation with the ARM template deployment.

Linked services deployment using the ARM template is not supported

Below is a feedback item regarding the same. This is open for the user community to upvote & comment on. This allows our product teams to effectively prioritize your request against our existing feature backlog and gives insight into the potential impact of implementing the suggested feature.

https://feedback.azure.com/d365community/idea/48f1bf78-2985-ec11-a81b-6045bd7956bb

As a workaround, please follow the below two-step process.

1. Set up a blank workspace, the default storage using ARM template
2. Once you create the initial workspace, set up git integration for the artifacts deployment(linked services, pipelines, SQL sripts, notebooks etc) (or) use the Synapse REST API or PowerShell

Refer to this document that provides guidance on how to use the API.
https://learn.microsoft.com/en-us/powershell/module/az.synapse/set-azsynapselinkedservice?view=azps-7.1.0

Please note:

1. Spark pools and self-hosted integration runtimes aren't created in a workspace deployment task. If you have a linked service that uses a self-hosted integration runtime, manually create the runtime in the new workspace.
2. If the items in the development workspace are attached to the specific pools, make sure that you create or parameterize the same names for the pools in the target workspace in the parameter file.
3. If your provisioned SQL pools are paused when you attempt to deploy, the deployment might fail.

CICD documentation: https://learn.microsoft.com/en-us/azure/synapse-analytics/cicd/continuous-integration-delivery#azure-synapse-analytics

Other prerequisites: https://learn.microsoft.com/en-us/azure/synapse-analytics/cicd/continuous-integration-delivery#other-prerequisites

The below "template" creates storage account, SQL account, and workspace.

https://learn.microsoft.com/en-us/azure/synapse-analytics/quickstart-deployment-template-workspaces

I hope this helps. Please let me know if you have any further questions.

If this answers your question, please consider accepting the answer by hitting the Accept answer button, as it helps the community.

{  
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",  
    "contentVersion": "1.0.0.0",  
    "parameters": {  
        "workspaceName": {  
            "type": "String",  
            "metadata": {  
                "description": "The name of the Workspace."  
            }  
        },  
        "storageAccountName": {  
            "type": "String",  
            "metadata": {  
                "description": "The primary ADLS Gen2 Storage Account name. If not provided, the workspace name will be used."  
            }  
        },  
        "sqlAdministratorLogin": {  
            "type": "String",  
             "metadata": {  
                "description": "Provide the user name for SQL login."  
            }  
        },  
        "sqlAdministratorPassword": {  
            "type": "SecureString",  
            "metadata": {  
                "description": "The passwords must meet the following guidelines:<ul><li> The password does not contain the account name of the user.</li><li> The password is at least eight characters long.</li><li> The password contains characters from three of the following four categories:</li><ul><li>Latin uppercase letters (A through Z)</li><li>Latin lowercase letters (a through z)</li><li>Base 10 digits (0 through 9)</li><li>Non-alphanumeric characters such as: exclamation point (!), dollar sign ($), number sign (#), or percent (%).</li></ul></ul> Passwords can be up to 128 characters long. Use passwords that are as long and complex as possible. Visit <a href=https://aka.ms/azuresqlserverpasswordpolicy>aka.ms/azuresqlserverpasswordpolicy</a> for more details."  
            }  
        },  
        "tagValues": {  
            "defaultValue": {"Created with":"Synapse Azure Resource Manager deploment template"},  
            "type": "Object"  
        },  
        "cmkUri": {  
            "type": "String",  
            "defaultValue": "",  
            "metadata": {  
                "description": "The uri to a key in your Key Vault to add a second layer of encryption on top of the default infrastructure encryption"  
            }  
        },  
         "setSbdcRbacOnStorageAccount": {  
            "type": "bool",  
            "defaultValue": false  
        }  
    },  
    "variables": {  
        "dataLakeStorageName": "[if(empty(parameters('storageAccountName')), replace(tolower(parameters('workspaceName')),'-',''), tolower(parameters('storageAccountName')))]",  
        "cmkUriStripVersion": "[if(empty(parameters('cmkUri')), '', substring(parameters('cmkUri'), 0, lastIndexOf(parameters('cmkUri'), '/')))]",  
        "withCmk": {  
            "cmk": {  
                "key": {  
                    "name": "default",  
                    "keyVaultUrl": "[variables('cmkUriStripVersion')]"  
                }  
            }  
        },  
        "encryption": "[if(empty(parameters('cmkUri')), json('{}'), variables('withCmk'))]"  
    },  
    "resources": [  
        {  
            "type": "Microsoft.Resources/deployments",  
            "apiVersion": "2018-05-01",  
            "name": "storage",  
            "properties": {  
              "mode": "Incremental",  
              "templateLink": {  
                "uri": "[uri(deployment().properties.templateLink.uri, '/Azure-Samples/Synapse/master/Manage/DeployWorkspace/storage/azuredeploy.json')]",  
                "contentVersion": "1.0.0.0"  
              },  
              "parameters":{  
                "storageAccount":{"value": "[variables('dataLakeStorageName')]"},  
                "workspaceContainer":{"value": "[tolower(parameters('workspaceName'))]"}  
              }  
            }  
        },  
        {  
            "type": "Microsoft.Resources/deployments",  
            "apiVersion": "2018-05-01",  
            "name": "workspace",  
            "properties": {  
              "mode": "Incremental",  
              "templateLink": {  
                "uri": "[uri(deployment().properties.templateLink.uri, '/Azure-Samples/Synapse/master/Manage/DeployWorkspace/workspace/azuredeploy.json')]",  
                "contentVersion": "1.0.0.0"  
              },  
              "parameters":{  
                "name":{"value": "[tolower(parameters('workspaceName'))]"},  
                "sqlAdministratorLogin":{"value": "[parameters('sqlAdministratorLogin')]"},  
                "sqlAdministratorPassword":{"value": "[parameters('sqlAdministratorPassword')]"},  
                "setSbdcRbacOnStorageAccount":{"value": "[parameters('setSbdcRbacOnStorageAccount')]"},  
                "defaultDataLakeStorageAccountName":{"value": "[variables('dataLakeStorageName')]"},  
                "defaultDataLakeStorageFilesystemName":{"value": "[tolower(parameters('workspaceName'))]"},  
                "tagValues":{"value": "[parameters('tagValues')]"},  
                "encryption":{"value": "[variables('encryption')]"}  
              }  
            },  
            "dependsOn": [  
                "storage"  
            ]  
        }  
    ],  
    "outputs": {}  
}

Answer 2

This is not answering the question! Sorry