question

ks567 avatar image
0 Votes"
ks567 asked KranthiPakala-MSFT commented

Any Limitation in tumbling window trigger

In ADF, We have first few pipelines run in sequence with dependencies and branches out at the end. Having said that, its our first experience to opt for tumbling window trigger option. is there any limitation in this option? what are the other options available for scheduling Data factory Pipelines in Azure.

azure-data-factory
· 4
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Appreciate if I get this scenario answered,

There are multiple master pipelines should be triggered with dependencies.
The First one leading the queue is significant and has this criteria,
1) It has to check for a file arrival in FTP once in two weeks say every 15th and 30th of the month.
2) The file is not guaranteed to be exactly available on 15th/30th of every month, it could be 14 or 16th for 15th schedule and similarly for 30th.
3) Rest of the Trigger should follow the First event based trigger.
4) if any failure among triggers, either re-run or cancel the entire cycle (eg., 15th cycle)

0 Votes 0 ·

Hi anonymous user,

Sorry for the delay in response. Since there is no guarantee of the file arrival, I would suggest you to please check on the Schedule trigger option. You can start the schedule from 14th to 17th and similar for 30th to trigger your master pipeline once a day, and in the very beginning of the flow, have a validation activity to check if the file has arrived, if the file is arrived continue executing the flow, if not just end the flow. In continuation flow have execute pipeline activities in it to trigger the dependent pipeline in order once the prior activity is completed successfully. If any activity of the pipeline fails, then the subsequent flow is discontinued and the pipeline status is marked as failed.


0 Votes 0 ·

Continuation to above comment:

In case if you want to re-run from the failed activity, you can always go to Monitor section in ADF UI then to Pipeline runs and then click select the failed pipeline and you see an option Rerun from failed activity

82573-image.png

Hope this info helps.

Thanks


0 Votes 0 ·
image.png (45.9 KiB)
Show more comments

1 Answer

KranthiPakala-MSFT avatar image
0 Votes"
KranthiPakala-MSFT answered KranthiPakala-MSFT commented

Hi anonymous user,

Welcome to Microsoft Q&A forum and thanks for your query.

Tumbling window triggers are a type of trigger that fires at a periodic time interval from a specified start time (basically when you want to process a time based data), while retaining state. Tumbling windows are a series of fixed-sized, non-overlapping, and contiguous time intervals.

This triggers offer a suite of features for complex scenarios like :

Here are few Limitations:

  • A tumbling window trigger has a one-to-one relationship with a pipeline and can only reference a singular pipeline.

  • If one of the dependencies triggers fails, you must successfully rerun it in order for the dependent trigger to run.

  • A tumbling window trigger will wait on dependencies for seven days before timing out. After seven days, the trigger run will fail


ADF also has another time based trigger called Schedule trigger. To know more in detail about the differences between a Schedule trigger and a Tumbling window trigger, please refer to this doc: Trigger type comparison

78876-image.png

Related helpful docs: Tumbling window trigger Usage scenarios and examples

Hope this info helps.



Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.



image.png (91.8 KiB)
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi anonymous user,

Just checking in to see if the above suggestion was helpful. If this answers your query, please do click “Accept Answer” and/or Up-Vote, as it might be beneficial to other community members reading this thread. And, if you have any further query do let us know.

Thanks

0 Votes 0 ·