ARM template deployment on DevOps and Event triggers

HannaBel 1 Reputation point
2022-02-08T16:55:37.453+00:00

We have a CI/CD process set up to deploy ADF changes. Deployment with ARM Templates requires ADF triggers to be disabled and then enabled back. The time when triggers are disabled during deployment is about 5 minutes. We have some processes that rely on Event based triggers and if the file arrives during the time triggers are disabled, it is not being processed and technically lost which is critical if this file is for a dataset that we load incrementally.

What are the possible solutions here?

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
6,724 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. ShaikMaheer-MSFT 30,506 Reputation points Microsoft Employee
    2022-02-14T15:55:04.707+00:00

    Hi @HannaBel ,

    Thank you for posting query in Microsoft Q&A Platform.

    Technically, while trigger is disabled event based trigger will loose files which are created in that gap.

    We can consider below work around to process missed files if any.

    • When ever, we process file keep log of it in some log or audit table.
    • Run a pipeline to process missed files. This pipeline should take all file names from storage and then see which are missing in table. and then consider processing that files alone.

    Hope this helps. Please let us know if any further queries.

    ------------

    Please consider hitting Accept Answer button. Accepted answers helps community as well.