Azure Data Factory - Dynamic Rest URL

Mctwt 1 Reputation point
2020-07-03T02:45:56.983+00:00

Hi,

I'm trying to make a Azure Data Factory pipeline where First GET method is used to get all of specific filtered processings(done)

-Additonally how I can take only newest record for each processing ?

Next I need to pass those filtered processings (or activity output of previous pipeline element with output.item.processing)

for another GET.

-Dynamic URL for each element ?/All of outputs would be needed

After second get I should have complete information regarding those specific processing&partitions to provide to Pipeline which needs proces name and paritions passed (perhaps as parameter)

How this can be achieved in Azure Data Factory ?

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,122 questions
0 comments No comments
{count} votes

3 answers

Sort by: Most helpful
  1. ChiragMishra-MSFT 956 Reputation points
    2020-07-09T06:23:58.76+00:00

    Hi @Mctwt-8188,

    Sorry for the delayed response. There are a few ways to "incrementally copy" (Copy only new/updated records) depending on your source and sink in ADF.Assuming that your web activity gives you an array of elements to be processed, you can do the following :

    • Chain a For Each activity to iterate through the array.
    • Inside the ForEach activity, you can have items as something like (assuming array of items as outputArray) : @activity('MainWebActivity').output.outputArray
    • Inside the ForEach activity, you can have a parameterized web activity to iterate through each item in the array. Each item can be accessed using the expression :
      @item()

    Please look at this screen recording for details .

    PS - If you want to parameterize a REST based Linked Service and dataset, you can do so by following this article Hope this helps.


  2. Mctwt 1 Reputation point
    2020-07-09T08:52:59.333+00:00

    Hi Chirag,

    Thank you for answer, it's helpful.

    I will try describe it a bit detailed after recent trys to make it work.

    1. I made first GET which returns sepcific ProcessRunKey for each ProcessKey ( it's unsorted and unfiltered - failed to make it descending by date)

    Would I be able to use Filter Activity to take only record for specific list Of ProcessKey's ? and for example only take last 3 ProcessRunKeys for each ProcessKey ? Confused how to achieve that as my REST API seems to have problem with filtering in URL, due to that would need to use Filter Activity

    1. In second GET i Pass output.item.ProcessKey from first GET, which returns me new field runobjectkey that I need to use for third GET.
    2. With second GET finished I can run third GET passing runobjectkey as parameter.

    Now I would need to dynamically pass output of Third Get (Object & partitions - there are few for each runobjectkey)
    table to dfiferent parameter and partitions to different Parameter ( Do i need loops or should i pass it as array to containt all results and then for each results use Loop to pass parameters next)

    Next I would probably use loop to keep passing Object, Partition to pipeline which triggers ETL based on those values


  3. MartinJaffer-MSFT 26,061 Reputation points
    2020-07-20T20:49:16.657+00:00

    @mctwt-8188

    Filter activity can be used to take only records with processkey which belong to a known set.

    Example:
    Pipeline with 2 array parameters:
    13103-subsets.jpg
    And a filter activity
    13121-subsets2.jpg

    tests each input item and only accepts if it is part of the other set.
    Yours will look more complex, because you are testing a property, not simple numbers like this example.

    0 comments No comments