Azure Data Factory - Dynamic Rest URL

Question

Hi,

I'm trying to make a Azure Data Factory pipeline where First GET method is used to get all of specific filtered processings(done)

-Additonally how I can take only newest record for each processing ?

Next I need to pass those filtered processings (or activity output of previous pipeline element with output.item.processing)

for another GET.

-Dynamic URL for each element ?/All of outputs would be needed

After second get I should have complete information regarding those specific processing&partitions to provide to Pipeline which needs proces name and paritions passed (perhaps as parameter)

How this can be achieved in Azure Data Factory ?

Answer 1

Hi @Mctwt-8188,

Sorry for the delayed response. There are a few ways to "incrementally copy" (Copy only new/updated records) depending on your source and sink in ADF.Assuming that your web activity gives you an array of elements to be processed, you can do the following :

• Chain a For Each activity to iterate through the array.
• Inside the ForEach activity, you can have items as something like (assuming array of items as outputArray) : @activity('MainWebActivity').output.outputArray
• Inside the ForEach activity, you can have a parameterized web activity to iterate through each item in the array. Each item can be accessed using the expression :
  @item()

Please look at this screen recording for details .

PS - If you want to parameterize a REST based Linked Service and dataset, you can do so by following this article Hope this helps.

Answer 2

Hi Chirag,

Thank you for answer, it's helpful.

I will try describe it a bit detailed after recent trys to make it work.

1. I made first GET which returns sepcific ProcessRunKey for each ProcessKey ( it's unsorted and unfiltered - failed to make it descending by date)

Would I be able to use Filter Activity to take only record for specific list Of ProcessKey's ? and for example only take last 3 ProcessRunKeys for each ProcessKey ? Confused how to achieve that as my REST API seems to have problem with filtering in URL, due to that would need to use Filter Activity

2. In second GET i Pass output.item.ProcessKey from first GET, which returns me new field runobjectkey that I need to use for third GET.
3. With second GET finished I can run third GET passing runobjectkey as parameter.

Now I would need to dynamically pass output of Third Get (Object & partitions - there are few for each runobjectkey)
table to dfiferent parameter and partitions to different Parameter ( Do i need loops or should i pass it as array to containt all results and then for each results use Loop to pass parameters next)

Next I would probably use loop to keep passing Object, Partition to pipeline which triggers ETL based on those values

Answer 3

@mctwt-8188

Filter activity can be used to take only records with processkey which belong to a known set.

Example:
Pipeline with 2 array parameters:

And a filter activity

tests each input item and only accepts if it is part of the other set.
Yours will look more complex, because you are testing a property, not simple numbers like this example.