ADF Pipeline concurrency for different pipelines

Monica Manoharan 66 Reputation points
2022-01-17T10:30:59.37+00:00

Hello Team,

I am having two different pipelines. Both the pipelines are configured to process data and insert in to a same table. So when these pipelines run parallelly, sometimes the first pipeline already inserts the data and the second pipeline has already taken a snapshot of the table even before the first pipeline's insertion and tries to insert again causing an unique index violation in the table.

So, is there a way to avoid this and take the latest snapshot before inserting the data ?

Elsewhere, I am trying to run the pipelines sequentially. So the concurrency parameter prevents from running the same pipeline again. So, is there a way to manage the ADF to entirely run one pipeline at a time irrespective of the different pipelines ?

Thanks,
Monica.M

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,609 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. ShaikMaheer-MSFT 37,896 Reputation points Microsoft Employee
    2022-01-18T13:31:10.457+00:00

    Hi @Monica Manoharan ,

    Thank you for posting query in Microsoft Q&A Platform.

    Making sure running pipelines in sequence is best option in your case.

    You can create a master pipeline which will run all your ADF pipelines in Sequence order. You can leverage Execute Pipeline activity for running pipeline.

    You can also consider writing your own code which will run pipelines in ADF sequential one by one by making REST API calls. In Azure, there is a service called Azure function in which you can deploy your code to run.
    Click here for ADF REST APIs documentation.
    Click here for Azure Function documentation.

    By recommendation will be considering having master pipeline to run your ADF pipelines.

    Hope this will help. Please let us know if any further queries.

    ---------------

    Please consider hitting Accept Answer button. Accepted answers helps community as well.