ADF Tumbling Window Trigger how to limit Queuing of pipeline runs?

Question

Currently I have a tumbling window trigger which triggers a pipeline hourly, and the pipeline concurrency is set to 1. If the pipeline is already running for more than 1 hour the subsequent triggers (triggered every hour) get queued up and eventually start failing after 100. Is there a way not to queue the tumbling window trigger if "X" number of triggers are already queued up?

Answer 1

Hello @Keerthi ,
Thanks for the question and using MS Q&A platform.

There are two places where concurrency is set. You already found the pipeline concurrency, but there is also a tumbling window trigger concurrency.

Therevis another thing to try, I assume your trigger currently does not have any dependencies. If you set up self-dependency, that is make the trigger depend upon the success of the previous run, then the trigger has no choice but to wait for completion before adding to the queue.

Please do let me if you have any queries.

Thanks
Martin

---

• Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
  • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

Answer 2

Thanks @MartinJaffer-MSFT , previously we used the self-dependency in the trigger but that created issues when re-triggering a failed trigger. We were not able to consistently re-trigger a failed trigger and the only work around was to remove the self-dependency and manually re-triggering for the failed windows. ADF team had mentioned there was a bug which prevented re-triggering with self-dependency when trigger max concurrency is set to 1 and they had no ETA for the fix. Hence, we moved away from trigger self-dependency, but we have this queuing issue now.