CI/CD Setup for Git-Integrated Synapse Workspaces with Azure DevOps – Best Practices and Team Collaboration (Synapse Live Mode)
Hello! I’ve implemented a full CI/CD deployment system using an artifacts transformer and a Deployment pipeline within Azure DevOps (sharing both code snippets below).
Artifacts Transformer Pipeline:
trigger:
branches:
include:
- dev
pool:
vmImage: 'ubuntu-latest'
variables:
- group: artifactsTransformer-config
steps:
- checkout: self
persistCredentials: true
fetchDepth: 0
- task: Bash@3
displayName: 'Copy and replace names in files'
env:
SYSTEM_ACCESSTOKEN: $(System.AccessToken)
inputs:
targetType: 'inline'
script: |
git config --global user.email "$(gitUserEmail)"
git config --global user.name "$(gitUserName)"
git remote set-url origin "$(repoUrl)"
git checkout dev
git pull origin dev
rm -rf $(qaFolder)
cp -r $(devFolder) $(qaFolder)
find $(qaFolder) -type f -exec sed -i "s/$(devFolder)/$(qaFolder)/g" {} +
git add $(qaFolder)
git commit -m "$(commitMessage)"
git push origin dev
git fetch origin
git checkout qa
git reset --hard origin/dev
git push --force origin qa
Deployment Pipeline:
trigger:
branches:
include:
- qa
pool:
vmImage: 'ubuntu-latest'
variables:
- group: goat-config
steps:
- task: Synapse workspace deployment@2
inputs:
operation: 'deploy'
TemplateFile: '$(Build.SourcesDirectory)$(templateFilePath)'
ParametersFile: '$(Build.SourcesDirectory)$(parametersFilePath)'
azureSubscription: '$(azureSubscription)'
ResourceGroupName: '$(resourceGroupName)'
TargetWorkspaceName: '$(workspaceName)'
DeleteArtifactsNotInTemplate: true
DeployManagedPrivateEndpoints: false
FailOnMissingOverrides: false
Environment: 'prod'
npmpackage: 'prod'
Looking for validation and best practices regarding the setup of a Git-integrated Synapse workflow between workspaces. Does the following approach seem correct? If not, what could be improved?
Additionally, in a scenario where other team members need to switch to the Git-integrated branch, the live Synapse workspace would no longer display the resources, correct?
Would this require migrating all artifacts from the live workspace to the Git branch as a one-time setup? If so, what is the proper and recommended way to perform this migration?
Any insights, recommendations, or shared experiences would be greatly appreciated.