Synapse pipeline debug no longer running concurrent foreach loops

Cameron 46 Reputation points
2021-10-13T00:13:37.24+00:00

I just kicked off a debug run of a pipeline we use to load up a list of tables, but for the first time I got a message stating -

Starting pipeline <pipelinename> (Pipeline). All the activities inside the foreach loop will be run sequentially and each run pipeline activity will wait on completion for debugging purposes. For triggered runs, the foreach loop will use the defined ‘Batch count’ for parallel runs.

I've never received this before, in the past all foreach loops would run in the batch count specified, has something changed here?

Azure Synapse Analytics
Azure Synapse Analytics
An Azure analytics service that brings together data integration, enterprise data warehousing, and big data analytics. Previously known as Azure SQL Data Warehouse.
4,395 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. ShaikMaheer-MSFT 37,896 Reputation points Microsoft Employee
    2021-10-13T14:10:07.793+00:00

    Hi @Cameron ,

    Thank you for posting your query here.

    I tried ForEach by passing an array in batch fashion. I have not encountered any error in debug mode.

    Could you check how many Items your array contains and what is batch count defined?

    Please note, The ForEach activity has a maximum batchCount of 50 for parallel processing, and a maximum of 100,000 items.

    Click here to know about limitations of ForEach.