Triggering the pipeline failed due to large run size

Ryan Abbey 1,171 Reputation points
2021-09-23T00:05:06.38+00:00

A pipeline that ran yesterday, hasn't been changed since is now failing with "Triggering the pipeline failed due to large run size"... is there a limit to the number of pipelines/size of other running pipelines that can cause this? I do have another process running that is looping a large number of files, is that likely to be a cause? If so, what can be done to allow another pipeline to run at the same time?

If not, why is Data Factory so unstable that a pipeline sometimes has an issue with "Triggering the pipeline failed due to large run size"?

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,539 questions
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. ShaikMaheer-MSFT 37,896 Reputation points Microsoft Employee
    2021-09-23T08:41:54.1+00:00

    Hi @Ryan Abbey ,

    Thank you for posting query on Microsoft Q&A Platform.

    This might be because of your ADF may crossed some limitations. Below image shows few. Click here to see full list of ADF limitations.

    134622-image.png

    Please let us know if that helps or any further queries. Thank you.