How to implement a dynamic trigger interval in ADF without creating infinite loops or pipeline concurrency issues?

Question

Hello everyone,

I am currently working on optimizing an existing data pipeline in Azure Data Factory (ADF) and I am facing an architectural challenge regarding dynamic scheduling.

Here is my current setup:

• The pipeline is currently triggered by a fixed Schedule Trigger every 15 minutes.
• I have recently introduced a Databricks Notebook activity in the middle of the flow to check if data optimization is required. This notebook outputs a boolean status (True or False).

I then placed an If Condition activity right after the notebook.

My goal:

• If the notebook returns False (normal state): The pipeline should run its normal activities and be triggered again 15 minutes later.
• If the notebook returns True (optimization needed): The data retrieval frequency needs to slow down, and the pipeline should only be triggered 1 hour later (instead of 15 minutes).

The issue I am facing: I initially tried using Wait activities inside my If Condition branches, but I realized this does not affect the main Schedule Trigger, which keeps forcing a new pipeline run every 15 minutes regardless. Additionally, ADF natively blocks the pipeline from calling itself via an Execute Pipeline activity due to recursive cycle validation errors.

I want to avoid pipeline overlapping/concurrency bottlenecks and optimize Azure orchestration costs (avoiding keeping an ADF pipeline active with a long Wait activity).

What are the best practices or recommended architectures in Azure to handle this type of dynamic trigger interval based on a notebook output? I was advised to use an ADF Web Activity to call an external Logic App (Consumption plan) with a dynamic Delay action to handle the wait outside of ADF and then re-trigger the pipeline. Does this sound like the best approach, or is there a cleaner native alternative?

Thank you in advance for your help and insights!

Answer 1

Hi @Fasd
ADF Schedule triggers are static and cannot dynamically change their interval based on activity outputs. Therefore, a Wait activity or Databricks notebook result will not affect the next scheduled execution.

Setting pipeline concurrency to 1 can prevent overlapping executions, but it does not achieve the requirement of changing the trigger frequency from 15 minutes to 1 hour; it only queues additional runs.

There is no native ADF mechanism to dynamically reschedule the next execution based on runtime output. The recommended approaches are:

• Use two separate schedule triggers (15-minute and 1-hour) and conditionally skip executions based on the notebook output (preferred native approach).
• Use an external orchestrator such as Logic Apps or Azure Functions to control when the next pipeline execution should occur if truly dynamic scheduling is required.

I would not recommend long Wait activities, recursive pipeline execution patterns, or relying solely on concurrency settings, as these do not solve the dynamic scheduling requirement.

Answer 2

Azure Data Factory schedule triggers do not support dynamically changing the next fire time based on pipeline output. A schedule trigger’s recurrence (frequency/interval/schedule) is fixed in the trigger definition and is evaluated independently of pipeline logic.

Key behaviors from schedule triggers:

• Recurrence is defined by frequency, interval, and optional schedule (hours, minutes, weekDays, etc.).
• The scheduler computes future occurrences from startTime and recurrence; it does not read pipeline outputs.
• When hours/minutes are not set in the schedule, the first execution time is used as default for subsequent runs.
• Schedule triggers cannot be used for backfill and do not have retry or concurrency controls like tumbling window triggers.

Because of this, the following patterns apply:

1. A single schedule trigger cannot be made “dynamic” based on notebook output.
2. Using Wait inside the pipeline does not affect the trigger; it only keeps the current run alive.
3. Self-calling pipelines via Execute Pipeline are blocked by design to avoid recursive cycles.

Within native ADF capabilities, the supported way to vary frequency is to define different triggers with different recurrences and control which ones are active:

• For example, one trigger with frequency = minute, interval = 15 and another with frequency = hour, interval = 1.
• Each trigger can also use a schedule object to further constrain when it runs (e.g., specific minutes/hours or weekdays) as shown in the recurrence examples:
  • {"minutes":[0,15,30,45]} → every 15 minutes.
  • {"minutes":[0], "hours":[6], "monthDays":[1,-1]} → specific days/times.

However, trigger activation/deactivation is not driven by pipeline output; it is a management operation. There is no native mechanism in ADF for a pipeline to change its own trigger’s recurrence or switch between triggers based on runtime logic.

Given the documented behavior, implementing a dynamic delay outside ADF (for example, via an external orchestrator that can:

• Receive a signal from the pipeline (e.g., via Web activity),
• Apply a dynamic delay, and
• Then invoke the pipeline again

is consistent with the platform’s design, because it moves the “dynamic schedule” logic into a system that supports programmatic delays and callbacks. ADF itself only supports static recurrence definitions and cannot avoid overlapping runs except by design (e.g., ensuring the pipeline completes within the fixed interval or using a different trigger type such as tumbling window when backfill and strict windowing are required).

There is no cleaner fully native alternative in ADF documented for dynamically changing the next trigger time based on pipeline output; solutions rely on fixed recurrences and external orchestration if dynamic behavior is required.

---

References:

• Create a trigger that runs a pipeline on a schedule
• Pipeline execution and triggers in Azure Data Factory or Azure Synapse Analytics