ADF Until activity slow to terminate

Question

I have a number of ADF pipelines that I'm using to download historic transactional data from various external REST APIs. Because of the data volumes involved, the pipelines I've created run for many hours.

The API pagination mechanisms are such that I can't use the pagination rules built in to the Copy activity. I've therefore had to put a Copy activity (with no pagination rules) into a suitably crafted loop using an Until activity.

When I run these pipelines I can see that the Until activity isn't completing in a timely manner, despite the exit criteria being met and no further inner activities being executed. The delay varies from 20-30 minutes up to as much as several hours.

At the risk of stating the obvious, I expect pipelines to complete as soon as their work is done but this isn't what's happening. I can see that this issue was raised more than a year ago, but was never resolved.

Answer 1

You may need to verify who is the Until activity configuration is done the exit criteria are correctly set. You can enable detailed logging and monitoring to identify where delays occur and verify that inner activities are optimized.

Investigate the performance of the external REST APIs, as they might be causing the delays.

Review the timeout settings for the Until activity and consider breaking down the process into smaller, more manageable chunks.

If the problem persists, contact Microsoft support for further assistance.

Answer 2

Thanks for the feedback, though it's difficult seeing how any of your suggestions apply. I can see that the exit criteria of the Until activity has been met, but even though none of the inner activities are running there's a big delay exiting the loop.

Because no inner activities are running, this can't be an issue with the Web activities and how they issue REST API requests. I can't reduce the number of inner activities either, as everything is needed in order to meet the required pagination behaviour.

Unless I've configured something wrongly (which doesn't appear to be the case), this is a significant problem, that makes ADF unsuitable as an enterprise solution. This issue was previously raised more than a year ago, but appears to have been simply ignored.

Answer 3

Experiencing the same Until Loop exit delay problem with a very similar implementation (API with pagination in a loop).

For a completed run, there was ~26 minute delay.

Caught it in action and the next/final activity in the pipeline isn't queued or anything... just the loop activity in progress.