Synapse pipeline trigger one pipeline AFTER another in Azure Pipelines

Question

Synapse pipeline trigger one pipeline AFTER another in Azure Pipelines

NC 31

Hi,
I would like to create a task chain in Azure Synapse but I can not find that solution in the internet or in any documents from Microsoft.
I have two synapse workspaces A and B. Workspace A will copy files to data lake and Workspace B will read those files from data lake and process them. I assume that in this step I can use "Event based trigger" and specify the container name. Please correct me if I am wrong. I assume that this trigger will read all the new files that created in the container that I specific earlier.

Then In work space B, I have 5 pipelines that I want to create a task chain trigger that runs sequentially. How can I do that?

HimanshuSinha-msft 19,491 Reputation points Microsoft Employee Moderator

2022-04-27T19:28:16.32+00:00

Hello @NC ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, will respond back with the more details and we will try to help .
Thanks
Himanshu

1 answer

Your answer

HimanshuSinha-msft 19,491 Reputation points Microsoft Employee Moderator

2022-04-27T19:28:16.32+00:00

Hello @NC ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, will respond back with the more details and we will try to help .
Thanks
Himanshu

Answer 1

Vidya Narasimhan 2,126 Microsoft Employee

@NC yes, you can trigger a Synapse pipeline based on a Blob storage event like ‘blob created’ . For every new blob created matching the pattern/configuration that you have specified, a new event is generated that will trigger your pipeline . More details here https://learn.microsoft.com/en-us/azure/data-factory/how-to-create-event-trigger?tabs=data-factory
To run Synapse pipeline sequentially, one of the way is to add the pipelines as an activity and then define whether the pipeline waits for another dependent pipeline to execute using waitoncompletion parameter https://learn.microsoft.com/en-us/azure/data-factory/control-flow-execute-pipeline-activity

Jerome Rougnon-Glasson 1 Reputation point

2024-11-26T07:52:55.7+00:00

Both proposed solutions are inadequate:

Using a file trigger: This approach is often impractical, as most pipelines either write multiple files or none at all. In cases where multiple files are written, pipeline B would be triggered multiple times, breaking transactional consistency. Additionally, relying on file creation is not a clean or robust mechanism for inter-pipeline dependency.

Creating a top-level pipeline AB to trigger pipelines A and B: While functional, this solution is not equivalent to having a dedicated trigger that listens for the completion of pipeline A. This approach combines multiple pipelines into a single monolithic pipeline, which can become cumbersome to maintain. Encapsulating responsibilities into autonomous, modular pipelines is generally a better practice to ensure maintainability and flexibility.

Additional Note: Azure DevOps pipelines natively support pipeline triggers, which are a more appropriate and elegant solution for such use cases.
Jerome Rougnon-Glasson 1 Reputation point

2024-11-26T07:53:53.1933333+00:00

Both proposed solutions are inadequate:

Using a file trigger: This approach is often impractical, as most pipelines either write multiple files or none at all. In cases where multiple files are written, pipeline B would be triggered multiple times, breaking transactional consistency. Additionally, relying on file creation is not a clean or robust mechanism for inter-pipeline dependency.

Creating a top-level pipeline AB to trigger pipelines A and B: While functional, this solution is not equivalent to having a dedicated trigger that listens for the completion of pipeline A. This approach combines multiple pipelines into a single monolithic pipeline, which can become cumbersome to maintain. Encapsulating responsibilities into autonomous, modular pipelines is generally a better practice to ensure maintainability and flexibility.

Additional Note: Azure DevOps pipelines natively support pipeline triggers, which are a more appropriate and elegant solution for such use cases.

Share via

Synapse pipeline trigger one pipeline AFTER another in Azure Pipelines

1 answer

Your answer