Dynamically fetch a "since" Timestamp in the body of my web activity and dynamically update it after successful pipeline runs

Question

Hi All,

I have been stuck with this issue - I have a pipeline - see attached a screenshot, that would would download tables from a REST API asynchronously. The pipeline has the below workflow:

Step 1: web activity to get access_token using a POST method. Here, API KEY was passed for the activity to return an acess_token

Step 2: Another web activity that would accept that access_token from step one using a POST method of API calls. Here, the body of the web activity contains this json - {"format":"jsonl","since":"2023-03-07T20:01:39.135Z"}

Step 3: An Until activity which contains another 'web activity' and a 'wait' condition. The web activity in step 2, would get a update query and the one inside the Until activity, would return the status (either waiting or complete). The 'Wait condition' tells the Until asctivity to either continue to the next task when the status is complete or wait until it is complete.

Step 4: Another web activity which will get url downloads that would be passed to a copy activity

Step 5: ForEach activity which contain a copy activity. This copy activity uses the web activity from step 4 to download a files from the api into a datalake.

Following the above workflow, I was able to get the pipeline working and some files were downloaded in my gen2 datalake.

Here is my challenge:

Because I would want to query this data almost on a daily, I want to be able to DYNAMICALLY reference the json in the body of the web activity in STEP 2.

For clarity, what I mean is, say I run this pipeline today, according to the body of the web acivity in STEP 2, I want to fetch file updates "since":"2023-03-07T20:01:39.135Z" to "2023-03-10T20:01:39.135Z". That would mean that, if I run the pipeline tomorrow, I would want to fetch files "since":"2023-03-11T20:01:39.135Z". What I want to achieve is the ability to dynamically fetch this without having to do manually update the json in the body of web activity in step 2. I want the body in STEP 2 to dynamically update to the new "since":"Timestamp" everytime the pipeline is run.

I would appreciate if any one has any idea on how to execute this.

Thanks Isaac Screenshot 2023-03-10 at 16.46.42.png

Answer 1

@Isaac Odeh Hello and welcome to Microsoft Q&A.

A side note before getting to the answers. The formatting ate your image. I tried to fix the formatting, but the image was already gone.

I'm thinking the "Tumbling Window Trigger" is what you need. From the trigger you can get "window start time" and "window end time". This is effectively your example datetimes. Windows are contiguous, the next one starting where the last one ended. First you set up a pipeline parameter, then when you set up the trigger for the pipeline, you assign to the parameter @trigger().outputs.windowStartTime Then you can do in step 2:

{"format":"jsonl","since":"@{pipeline().parameters.my_window_start_parameter}"}