ADF Data Flow: Possible to Organize Sink Order?

PageChristopher-5206 55 Reputation points
2023-10-17T14:24:22.42+00:00

Hi,

Is it possible to order sinks in such a way that one sink activity will not start until another sink activity has completed? Example below, the Hub record has a key that is a foreign key in the Sat record, so the Hub record has to exist before the Sat record can be added.

User's image

Thanks, Chris

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,711 questions
0 comments No comments
{count} votes

Accepted answer
  1. Suba Balaji 11,186 Reputation points
    2023-10-17T15:24:46.14+00:00

    Hi PageChristopher-5206

    Thanks for using MS Q&A platform and posting your query.

    You can leverage sink order under settings tab.

    Click anywhere outside the activities in data flow and you will see settings and in that custom sink ordering as below.

    Screenshot 2023-10-17 at 8.52.37 PM

    Hope that helps.


1 additional answer

Sort by: Most helpful
  1. KeshavKiran-MSFT 391 Reputation points Microsoft Employee
    2023-10-17T15:38:58.2033333+00:00

    Hi PageChristopher-5206,

    Thanks for the question and using MS Q&A platform.

    By default, data is written to multiple sinks in a nondeterministic order. The execution engine writes data in parallel as the transformation logic is completed, and the sink ordering might vary each run.

    However as I understand your scenario where you have foreign key dependency and the hub record has to exist first before Sat record can be inserted, you can leverage the Customer sink ordering functionality which we have in ADF.

    To specify an exact sink ordering, enable Custom sink ordering on the General tab of the data flow. When enabled, sinks are written sequentially in increasing order ( Sink Hub should have write order 1 and then Sink Set as 2).

    https://learn.microsoft.com/en-us/azure/data-factory/data-flow-sink#custom-sink-ordering

    sinkSat

    Please try this out and let us know in case of any questions.