Idle time on For Each File Activity

Matt Michalowski 26

I have a pipeline configured as shown below. It checks a subdirectory for a group of flat files and based on the name of the file, it will load data into a SQL staging table. The ForEach activity has 6 If Condition activities which will run based on the file name parameter that's passed.

On this particular run, all activities within the For Each complete in about 2 1/2 minutes. Then there's an additional 3 minutes while the For Each stays "in progress" before finally completing successfully. The Gannt chart is shown below. I don't understand why there appears to be 3 minutes of "idle" time after the last inner activity of the For Each completes and the entire For Each activity ends. Please advise.

HimanshuSinha-msft 19,381 Reputation points Microsoft Employee

2021-06-07T21:03:28.157+00:00

Hello @Matt Michalowski ,
Thanks for the ask and using the Microsoft Q&A platform .
At this time, we are reaching out to the internal team to get some help on this . We will update you once we hear back from them.

Can you please share the runid and the pipeline name ?
Thanks
Himanshu
HimanshuSinha-msft 19,381 Reputation points Microsoft Employee

2021-06-09T18:52:17.137+00:00

Hello @Matt Michalowski ,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, will respond back with the more details and we will try to help .

Thanks
Himanshu
HimanshuSinha-msft 19,381 Reputation points Microsoft Employee

2021-06-16T16:32:10.7+00:00

Hello @Matt Michalowski ,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, will respond back with the more details and we will try to help .

Thanks
Himanshu
Matt Michalowski 26 Reputation points

2021-11-19T22:25:35.06+00:00

I am still experiencing idle time issues in our pipelines that use For Each and Switch activities. I have an example from last night where a pipeline took 2 min 40 seconds to complete, but the bulk of the work within the pipeline (truncating a table, loading data from a file, and archiving the file) only takes a little over 30 seconds. It's similar to the issue described in the original post above. Here are the details:

Pipeline name: CMCMLS Master
Pipeline run ID: db1ae75d-834d-4b33-b3ae-41f28eec8b2e

Again, this is just one example. We have many pipelines running on a nightly basis, and all of this idle time in our For Each and Switch activities is adding up and becoming a major issue. If there is a setting or some other configuration that would make sense to eliminate or at least cut down on the wasted time within each pipeline, that would be very helpful.
Matt Michalowski 26 Reputation points

2021-12-01T16:01:48.497+00:00

Hi @HimanshuSinha-msft

I shared an example pipeline name and pipeline run ID in my previous message on 11/19/21. Is there any additional information I can provide to help get this resolved?

Share via

Idle time on For Each File Activity