Azure Synapse deployment pipeline with private endpoints

Question

I have two Synapse environments, DEV and UAT. I setup an Azure Pipeline with the following task

- task: Synapse workspace deployment@2
  displayName: "Synapse workspace deployment to Acceptance"
  inputs:
    operation: 'validateDeploy'
    azureSubscription: 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
    ResourceGroupName: '$(resourceGroupName)'
    TargetWorkspaceName: '$(synapseWorkspaceName)'
    DeleteArtifactsNotInTemplate: true
    DeployManagedPrivateEndpoints: false
    FailOnMissingOverrides: false
    Environment: 'prod'
    ArtifactsFolder: '$(System.DefaultWorkingDirectory)'
    TemplateFile: '$(System.DefaultWorkingDirectory)/synw-xx-xxxxxx-d/TemplateForWorkspace.json'
    ParametersFile: '$(System.DefaultWorkingDirectory)/synw-xx-xxxxx-d/TemplateParametersForWorkspace.json'
    OverrideArmParameters: >
      -ls_az_adls_saxxxxxxx_properties_typeProperties_url $(ls_az_adls_saxxxxxxx_properties_typeProperties_url)

The private endpoints were created automatically when I deployed the both Synapse instance. So I do not want to deploy the private endpoints of DEV to UAT, therefore I set

DeployManagedPrivateEndpoints: false

And I want artifacts that were deleted from DEV to also be deleted from UAT, therefore, I set

DeleteArtifactsNotInTemplate: true

When I run the pipeline, I get an error because it's trying to delete the private endpoints on my UAT environment because they are not defined in the template. Getting Artifacts which should be deleted from workspace.

Artifact not found in template. deleting synapse-ws-custstgacct--synw-xx-xxxxx-a-saxxxxxxxa of type Microsoft.Synapse/workspaces/managedVirtualNetworks/managedPrivateEndpoints

Found 1 artifacts in the workspace that many need to be deleted.

Computing dependancies for Artifacts which should be deleted from workspace.

Iteration 1 Figured out deletion order for 1 / 1 Artifacts for Dependancies.

Deleting synapse-ws-custstgacct--synw-xx-xxxxx-a-saxxxxxxxa of type managedPrivateEndpoints

For Artifact: synapse-ws-custstgacct--synw-xx-xxxxx-a-saxxxxxxxa: ArtifactDeletionTask status: 403; status message: Forbidden

deploy operation failed

This makes sense because the private endpoints are not named the same. I managed to deploy by setting DeleteArtifactsNotInTemplate to false, but that's not what I want.
I don't what the DEV private endpoints to be deployed in UAT and the UAT private endpoints to be deleted.

Is there a way to avoid this problem ?

Answer 1

When deploying Azure Synapse artifacts from a DEV environment to a UAT environment using Azure DevOps pipelines, managing private endpoints requires careful attention. The Synapse workspace deployment task includes parameters like DeployManagedPrivateEndpoints and DeleteArtifactsNotInTemplate to control the deployment behavior.

In your scenario, setting DeployManagedPrivateEndpoints to false ensures that private endpoints from DEV are not deployed to UAT. However, setting DeleteArtifactsNotInTemplate to true causes the pipeline to attempt deletion of artifacts in UAT that are not present in the DEV template, including managed private endpoints. This leads to a 403 Forbidden error when the pipeline tries to delete these endpoints, as they are not defined in the template and may have different names or configurations in UAT.

To address this issue, consider the following approaches:

Set DeleteArtifactsNotInTemplate to false: This prevents the pipeline from deleting any artifacts in UAT that are not present in the DEV template, including private endpoints. While this approach avoids deletion errors, it may leave obsolete artifacts in UAT.

Manually manage private endpoints: Before running the deployment pipeline, manually ensure that the necessary private endpoints exist in UAT and are correctly configured. This approach requires additional manual effort but provides precise control over private endpoint configurations.

Customize the deployment process: Modify the deployment pipeline to exclude private endpoints from the artifacts considered for deletion. This can be achieved by customizing the deployment scripts or using additional tasks to filter out private endpoints from the deletion list.

It's important to note that managed private endpoints are only supported in Synapse workspaces with a Managed workspace Virtual Network.

https://learn.microsoft.com/en-us/azure/synapse-analytics/security/synapse-workspace-managed-private-endpoints?utm_source=chatgpt.com

By carefully managing the deployment parameters and processes, you can ensure that private endpoints are appropriately handled during the deployment of Synapse artifacts between environments.