Until Activity Loop Exist consumes Lot of Time

Amol Rajmane 1 Reputation point
2022-06-11T14:59:41.57+00:00

In my pipeline Until activity run for at least 7-8 hours. But until activity takes more than 2 hours to come out of the loop. Please let me know any work around for the same.

Thanks
Amol

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,740 questions
{count} votes

2 answers

Sort by: Most helpful
  1. Amol Rajmane 1 Reputation point
    2022-06-13T03:10:31.403+00:00

    I have created a pipeline with "Until" activity to execute API call until API call return response code as 6. This response code is assigned to a variable. If the variable value is equal to 6 the loop will exit.

    Sometime API call returns the response code 6 with 2-3 minutes in that case "Until" activity takes 1 minute to exit the loop after receiving the response 6. Sometime API returns response code 6 after one hour but "Until" activity break the loop after 20 minutes. As the API call execute for more time the time taken to exit the loop also increases.

    Sometimes all API calls runs for 8 hours and even the last activity inside the loop successfully got executed, but the "Until" activity runs for another 2 hours without doing nothing. I hope I am able to explain the problem.

    0 comments No comments

  2. AnnuKumari-MSFT 32,991 Reputation points Microsoft Employee
    2022-06-13T12:04:48.32+00:00

    Hi @Amol Rajmane ,
    Welcome to Microsoft Q&A platform and thanks for posting your query.
    As I understand your issue, you are using web activity and set variable activity inside Until block. Once all the child activities are executed, we expect until activity to be completed and come out of the loop but same is not happening in your case. It takes lots of time to come out of the loop. Please let me know if my understanding is incorrect.

    Until activity inside ADF executes a set of activities in a loop until the condition associated with the activity evaluates to true. Your expectation is valid, it should break the loop once the conditions are met.

    My suggestion would be to monitor the pipeline for certain period of time and check what is the max time it is taking for child activities to get completed and accordingly , update the timeout value for until activity.

    Suppose, child activities take 2 hours maximum, then keep the timeout value of until activity as 140 mins or so having few minutes as buffer time.
    Also, consider adding wait activity as well inside until block so that API doesn't get stuck by continuous hit.

    Hope this will help. Please let us know if any further queries.

    ------------------------------

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you.
      Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.