CI/CD Powershell Azure Synapse Workspace with Azure Devops

Question

I am working on CI/CD between Azure Synapse Workspace with Azure Devops. 

I have source code from someone else who work on this project before and it works well and I try to understand it and create my own project on it. right now, I have issues of the after the deplolyment stage: 

after the deployment stage, it has a Powershell script for running synapse pipeline and the location for the script is in develop branch (which is CI). and this Powershell Script is also used in a pipeline yaml script in develop branch.

I know the yaml script is for running the pipeline with a powershell script.

my questions:

1. why do we need to embed a powershell script inside the pipeline yaml file? Could we just write the code for running a specific azure synapse workspace pipeline in the yaml file?
2. it seems like the powershell script is for passing parameters to the pipeline and what else is that doing?

thanks in advance

joan

Answer 1

I could be wrong but I think it is because synapse pipelines aren't directly supported as a task in devops pipelines so someone has used this as a workaround?
Your script just allows you to invoke a synapse pipeline with parameters and waits for it to complete.

There are some good posts (maybe getting a bit old now) on CI/CD for Synapse
https://techcommunity.microsoft.com/t5/data-architecture-blog/ci-cd-in-azure-synapse-analytics-part-1/ba-p/1964172

Hope this helps

Answer 2

Hello Zhuoqiong Mo

The PowerShell script is used to pass parameters to the Synapse pipeline, which is not possible with YAML alone. The PowerShell script is also used to perform other tasks such as setting up the environment, installing dependencies, and executing the pipeline. It is a good practice to keep the PowerShell script separate from the pipeline YAML file so that it can be reused across multiple pipelines and can be easily maintained.

In terms of writing the code for running a specific Azure Synapse Workspace pipeline in the YAML file, it is possible to do so using the Azure Synapse Workspace task in the YAML file. However, this task may not provide all the functionality that you need, which is why a PowerShell script is being used in this case.

I hope this helps! Let me know if you have any other questions.