Schedule a pipeline in Azure Synapse analytics

Question

How can I schedule a trigger pipeline in Azure Synapse analytics to run every 30 minutes, but avoid starting when another pipeline is running or at certain specified times (such as between 1 am to 3 am)? I am seeking guidance on the best approach for scheduling this pipeline.

Answer 1

There is a property called concurrency to avoid multiple runs of the same pipeline at same time.

Answer 2

Hi @Aditya Singh

Thanks for the question and using MS Q&A platform.

To schedule a trigger pipeline in Azure Synapse Analytics to run every 30 minutes while avoiding execution during certain times (like between 1 AM and 3 AM) or when another pipeline is running, you can use a combination of scheduling, conditional logic, and pipeline dependencies.

1. Schedule Trigger Configuration Create a schedule trigger within the Data Factory to initiate pipeline execution at predetermined intervals. Define the trigger's start date and recurrence pattern (e.g., every 30 minutes).

2. Conditional Pipeline Creation Construct a pipeline to evaluate the current time and determine subsequent actions. Introduce a pipeline parameter named 'currentTime' to store the current UTC time. Employ a 'Set Variable' activity to capture the current UTC time using the @utcnow() function and assign it to the 'currentTime' parameter. Utilize an 'If Condition' activity to evaluate if the current hour falls within a specified range (e.g., between 1 AM and 3 AM). The condition expression should compare the hour extracted from the 'currentTime' parameter against the desired range boundaries.

This expression checks if the current hour is between 1 AM and 3 AM.

3. Pipeline Branching Create two additional pipelines: 'DoNothing' and 'MainPipeline'. The 'DoNothing' pipeline can be empty or contain a placeholder activity (e.g., 'Wait'). The 'MainPipeline' encompasses the core data processing logic. Configure the 'If Condition' activity's branches: If the condition is true (current hour within the specified range), execute the 'DoNothing' pipeline. If the condition is false, execute the 'MainPipeline'.
4. Associate the Trigger with the Pipeline Associate the schedule trigger with the main pipeline: Navigate to the trigger created in Step 1. Click Edit. In the Pipelines section, add your main pipeline (MainPipeline) to the list. Save the trigger configuration.

This approach enables the pipeline to execute conditionally based on the current time. By leveraging a schedule trigger, the pipeline is initiated periodically. The pipeline's logic evaluates the current time and determines whether to proceed with the main data processing logic or perform no action (or a minimal operation) during specific hours. This conditional execution provides flexibility in orchestrating data processing tasks based on time-dependent requirements.
For reference please refer: https://www.youtube.com/watch?v=GCYfKDoKcek&t=1s

I hope this information helps. Please do let us know if you have any further queries.