CI/CD Setup for Git-Integrated Synapse Workspaces with Azure DevOps – Best Practices and Team Collaboration (Synapse Live Mode)

Question

Hello! I’ve implemented a full CI/CD deployment system using an artifacts transformer and a Deployment pipeline within Azure DevOps (sharing both code snippets below).

Artifacts Transformer Pipeline:

trigger: 
  branches:
    include:
      - dev

pool:
  vmImage: 'ubuntu-latest'

variables:
  - group: artifactsTransformer-config

steps:
  - checkout: self
    persistCredentials: true
    fetchDepth: 0

  - task: Bash@3
    displayName: 'Copy and replace names in files'
    env: 
      SYSTEM_ACCESSTOKEN: $(System.AccessToken)
    inputs:
      targetType: 'inline'
      script: |
        git config --global user.email "$(gitUserEmail)"
        git config --global user.name "$(gitUserName)"
        git remote set-url origin "$(repoUrl)"
        git checkout dev
        git pull origin dev
        rm -rf $(qaFolder)
        cp -r $(devFolder) $(qaFolder)
        find $(qaFolder) -type f -exec sed -i "s/$(devFolder)/$(qaFolder)/g" {} +
        git add $(qaFolder)
        git commit -m "$(commitMessage)"
        git push origin dev
        git fetch origin
        git checkout qa
        git reset --hard origin/dev
        git push --force origin qa

Deployment Pipeline:

trigger:
  branches:
    include:
      - qa

pool:
  vmImage: 'ubuntu-latest'

variables:
  - group: goat-config

steps:
  - task: Synapse workspace deployment@2
    inputs:
      operation: 'deploy'
      TemplateFile: '$(Build.SourcesDirectory)$(templateFilePath)'
      ParametersFile: '$(Build.SourcesDirectory)$(parametersFilePath)'
      azureSubscription: '$(azureSubscription)'
      ResourceGroupName: '$(resourceGroupName)'
      TargetWorkspaceName: '$(workspaceName)'
      DeleteArtifactsNotInTemplate: true
      DeployManagedPrivateEndpoints: false
      FailOnMissingOverrides: false
      Environment: 'prod'
      npmpackage: 'prod'

Looking for validation and best practices regarding the setup of a Git-integrated Synapse workflow between workspaces. Does the following approach seem correct? If not, what could be improved?

Additionally, in a scenario where other team members need to switch to the Git-integrated branch, the live Synapse workspace would no longer display the resources, correct?

Would this require migrating all artifacts from the live workspace to the Git branch as a one-time setup? If so, what is the proper and recommended way to perform this migration?

Any insights, recommendations, or shared experiences would be greatly appreciated.

Answer 1

Hi Napsty Dev,

The source workspace that's used for development configured with a Git repository in Azure Synapse Studio. https://learn.microsoft.com/en-us/azure/synapse-analytics/cicd/source-control#configuration-method-2-manage-hub

If you have an ARM template that deploys a resource, such as an Azure Synapse workspace, as the task performs to create or update infrastructure resources. https://learn.microsoft.com/en-us/azure/synapse-analytics/cicd/continuous-integration-delivery#set-up-a-stage-task-for-an-arm-template-to-create-and-update-a-resource

If a workspace not integrated with any git repo, the code is not present in branch, it is present in live mode. You can verify here how to move code from live mode to git branch.

If you have any queries, please do let us know, we will help you.