question

Cameron-0825 avatar image
0 Votes"
Cameron-0825 asked Cameron-0825 commented

Synapse pipeline debug no longer running concurrent foreach loops

I just kicked off a debug run of a pipeline we use to load up a list of tables, but for the first time I got a message stating -

Starting pipeline <pipelinename> (Pipeline). All the activities inside the foreach loop will be run sequentially and each run pipeline activity will wait on completion for debugging purposes. For triggered runs, the foreach loop will use the defined ‘Batch count’ for parallel runs.

I've never received this before, in the past all foreach loops would run in the batch count specified, has something changed here?

azure-synapse-analytics
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @Cameron-0825 ,

Following up to see you get chance to see above comment and share updates?

0 Votes 0 ·

1 Answer

ShaikMaheer-MSFT avatar image
0 Votes"
ShaikMaheer-MSFT answered Cameron-0825 commented

Hi @Cameron-0825 ,

Thank you for posting your query here.

I tried ForEach by passing an array in batch fashion. I have not encountered any error in debug mode.

Could you check how many Items your array contains and what is batch count defined?

Please note, The ForEach activity has a maximum batchCount of 50 for parallel processing, and a maximum of 100,000 items.

Click here to know about limitations of ForEach.



· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

There's 10 batch counts in this ForEach activity.

Seems to be, possibly, when the ForEach activity has another pipeline execution in it, but this wasn't the behaviour historically.

0 Votes 0 ·

Hi @Cameron-0825 ,

Since you are saying its not the case in past but now happening to be issue. I encourage you to have a support ticket for this and get deeper investigation. Thank you.

0 Votes 0 ·