CI/CD Powershell Azure Synapse Workspace with Azure Devops

Zhuoqiong Mo 20 Reputation points
2024-05-28T10:04:13.9833333+00:00

I am working on CI/CD between Azure Synapse Workspace with Azure Devops. 

I have source code from someone else who work on this project before and it works well and I try to understand it and create my own project on it. right now, I have issues of the after the deplolyment stage: 

after the deployment stage, it has a Powershell script for running synapse pipeline and the location for the script is in develop branch (which is CI). and this Powershell Script is also used in a pipeline yaml script in develop branch. User's image

User's image

I know the yaml script is for running the pipeline with a powershell script.

my questions:

  1. why do we need to embed a powershell script inside the pipeline yaml file? Could we just write the code for running a specific azure synapse workspace pipeline in the yaml file?
  2. it seems like the powershell script is for passing parameters to the pipeline and what else is that doing?

thanks in advance

joan

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,565 questions
{count} votes

2 answers

Sort by: Most helpful
  1. Michael Cameron 502 Reputation points
    2024-05-28T12:31:05.85+00:00

    I could be wrong but I think it is because synapse pipelines aren't directly supported as a task in devops pipelines so someone has used this as a workaround?
    Your script just allows you to invoke a synapse pipeline with parameters and waits for it to complete.

    There are some good posts (maybe getting a bit old now) on CI/CD for Synapse
    https://techcommunity.microsoft.com/t5/data-architecture-blog/ci-cd-in-azure-synapse-analytics-part-1/ba-p/1964172

    Hope this helps


  2. BhargavaGunnam-MSFT 28,271 Reputation points Microsoft Employee
    2024-05-29T21:48:31.72+00:00

    Hello Zhuoqiong Mo

    The PowerShell script is used to pass parameters to the Synapse pipeline, which is not possible with YAML alone. The PowerShell script is also used to perform other tasks such as setting up the environment, installing dependencies, and executing the pipeline. It is a good practice to keep the PowerShell script separate from the pipeline YAML file so that it can be reused across multiple pipelines and can be easily maintained.

    In terms of writing the code for running a specific Azure Synapse Workspace pipeline in the YAML file, it is possible to do so using the Azure Synapse Workspace task in the YAML file. However, this task may not provide all the functionality that you need, which is why a PowerShell script is being used in this case.

    I hope this helps! Let me know if you have any other questions.

    0 comments No comments