Synapse Pipelines ARM Template not working

Question

Hello!

I have a Synapse ARM Template that was generated by the "publish" option into a GIT repository, before moving to CI/CD DevOps deployment we are testing manually this ARM Template using the "Custom deployment" tool on the Azure Portal

We are facing the "Not found" error with just the simple creation of linked services and one test pipeline (with no dependencies between them), only the "AutoResolveIntegrationRuntime" item has no errors.

This is the error detail (exactly same error for the other 2)

We have tried a lot of changes on the ARM Template like renaming the name of the items, changing the API version to newer version, trying with different workspaces, creating the items on the target synapse workspace thinking that probably this process only updates instead of creating/update, and we did not have any luck.

Mi intuition is that the generation of the ARM Template by the "publish" option is not creating all the json properties in a correct way.

Attached the ARM Template we are using for this tests, pre-requisites are 1.- Synapse workspace, 2.- Create an azure key vault, 3.- Adls account.

Or if you want to same time you can actually leave the json parameters LS_adls and LS_keyvault as "default", the creation of the linked service object on synapse should not be validating if the target exists (its just the url property)

169497-synapse-simple-objects.txt

Answer 1

I have more news @AlejandroPion-9271 .

So I started comparing the template from the publish branch to the templates found in the "how to deploy synapse via arm" documentation page. I noticed a number of differences. First, there are parameters missing from the published version, such as the SQL Administrator password.
A closer look reveals the template from the document page contains under the "resource" section two "deployments". These "deployments" create the storage account and workspace itself .

All this points to a 2-step process. An initial one-time setup of things that will never change (Workspace name, default storage account, SQL administrator name). And subsequent deployment of artifacts. The template put out by "publish" is the latter of these two. This is further implied by directions in the CI/CD documentation page.

• Set up a blank workspace to deploy to:
  1. Create a new Azure Synapse workspace.

If you are coming from Azure Data Factory, I understand the confusion. If this was Data Factory, what you are doing would work. However Azure Synapse has more components which, due to having different origins, do not follow the same rules. This also explains why I couldn't find the "Export ARM Template" button in Azure Synapse; It doesn't exist.

So, in summary, if you want to continue your manual test, you will need to do 2 deployments. One to create the initial workspace, and a second using the stuff pulled from your GIT. After all, the CI/CD process is mostly about pushing updates.

Further caveats are called out in "Other prerequisites":

1. Spark pools and self-hosted integration runtimes aren't created in a workspace deployment task. If you have a linked service that uses a self-hosted integration runtime, manually create the runtime in the new workspace.
2. If the items in the development workspace are attached with the specific pools, make sure that you create or parameterize the same names for the pools in the target workspace in the parameter file.
3. If your provisioned SQL pools are paused when you attempt to deploy, the deployment might fail.

Hope this helps!
Martin

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how

Answer 2

After install the "Synapse workspace deployment" extension on your Azure DevOps, you can deploy the synapse "published" ARM Template through a yaml task like this:

- task: AzureSynapseWorkspace.synapsecicd-deploy.synapse-deploy.Synapse workspace deployment@1
  displayName: 'Azure Deployment: Create Or Update Synapse objects (Pipelines, LS, DS, Dataflows)'
  inputs:
    azureSubscription: '<Service Connection>'
    resourceGroupName: '<Resource Group Name>'
    TargetWorkspaceName: '<Synapse Workspace Name>'
    TemplateFile:  '<Folder Path>/Synapse.json'
    ParametersFile: '<Folder Path>/Synapse.param.dev.json'
    OverrideArmParameters: '
        -workspaceName "$(paramSynapse_WorkspaceName)" 
        -LS_sqlconfiguration_connectionString "Integrated Security=False;Encrypt=True;Connection Timeout=30;Data Source=$(paramSqlServerName).database.windows.net;Initial Catalog=$(paramSQLDbName)" 
        -mdmf-synapse-ws-dev-WorkspaceDefaultSqlServer_connectionString "Data Source=tcp:$(paramSynapse_WorkspaceName).sql.azuresynapse.net,1433;Initial Catalog=@{linkedService().DBName}" 
        -LS_adls_sink_properties_typeProperties_url "https://$(paramStorageAccountNameadls).dfs.core.windows.net/" 
        -LS_keyvault_properties_typeProperties_baseUrl "https://$(paramAzureKVName).vault.azure.net/" 
        -mdmf-synapse-ws-dev-WorkspaceDefaultStorage_properties_typeProperties_url "https://$(paramStorageAccountNameadls).dfs.core.windows.net"' 

Hope this can help someone else.
Regards,