Run Activity after "Until" Activity Times Out

Kaushik Mahadevan 0 Reputation points
2024-06-04T18:21:17.38+00:00

I have a pipeline containing an until activity that performs some web uploads and times out after 20 hours. If and when this activity times out, I want to be notified and run another activity. I'm aware that you can configure pipeline level notifications but specifically here, I want to run a logic app once the until activity times out. I also realize that I can implement some composition of pipelines but that seems like overkill for something this simple.

As it stands, when the until activity times out, the entire pipeline immediately fails. This mental model doesn't make sense to me, particularly in this use case, and I think the timeout should be specific to the activity and should be an activity level failure instead of a pipeline failure.

Has anyone arrived at any clean workarounds for this, albeit fairly niche, use case?

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,199 questions
{count} votes

1 answer

Sort by: Most helpful
  1. Amira Bedhiafi 20,176 Reputation points
    2024-06-04T19:45:29.7566667+00:00

    I have not experience this before but my idea is to configure your Until activity with a timeout setting (for example 20 hours) then you add a Fail activity or a Set Variable activity to be triggered upon failure of the Until activity. You can do this by setting up dependencies:

    • Select the Until activity, go to the General tab, and under the Activities section, configure the Success, Failure, and Completion dependencies.
    • Add a Set Variable activity or a Fail activity to handle the timeout case specifically.

    Then, you create a Logic App that will be triggered when a failure occurs, then use a Web Activity within your ADF pipeline to call the Logic App and configure it to send a POST request to your Logic App with necessary details.

    1 person found this answer helpful.
    0 comments No comments