There are substantial concurrent MappingDataflow executions which is causing failures due to throttling under Integration Runtime 'IR(Manage Private Network)',

Question

There are substantial concurrent MappingDataflow executions which is causing failures due to throttling under Integration Runtime 'IR(Manage Private Network)',

Adf2023 10

There are substantial concurrent MappingDataflow executions which is causing failures due to throttling under Integration Runtime.

Error code :4502.

PL was running last 3 months and suddenly consistently failing with above error. I tried use high IR config to increase capacity limit and separate IR only for this PL. Data size is very small and some case no data at all.

Here are the steps I took:

Increased IR capacity limit

2.Created new IR only for this PL

3.used different trigger times

4.tried with only debug session

None of the above workaround resolved this issue

KranthiPakala-MSFT 46,642 Reputation points Microsoft Employee Moderator

2023-02-01T23:22:39.95+00:00

Hi Adf2023,

As per the error description, seems like large number of Data Flow activity runs are occurring concurrently on the integration runtime.

Ref doc: Error code: 4502

Or may be hitting any of the below limitations. Azure Data Factory limits

Have you tried setting the pipeline concurrency to 1 and see if that helps to resolve the issue?

If the above checks don't help, I would recommend filing a support ticket for deeper investigation.

Do let us know how it goes.

Thanks
Adf2023 10 Reputation points

2023-02-02T14:55:44.41+00:00
@KranthiPakala-MSFT

Here is the config I have in IR which is dedicated to only this PL and Source (ADLS) not even have data to extract.

Yes, I set the pipeline concurrency to 1

Here is the overview of PL.

Export Data to ADLSGen2 by using Azure Synapse link push

DataVerse TO SQL ADF PL to load data from ADLS to Database(using Dataflow upsert method) I am wondering synapse link always actively pushing data to ADLS and when we run PL to extract data if that may cause issue. But i tried different trigger time and nothing resolved the issue.
KranthiPakala-MSFT 46,642 Reputation points Microsoft Employee Moderator

2023-02-03T22:35:03.0933333+00:00

Hi Adf2023,

Thanks for your response. If this issue persists consistently, then I recommend filing a support ticket for deeper investigation so that a support engineer can take a deeper look into this issue. Please do share the SR# once it is filed so that we can track it internally. In case if you don't have a support plan, please let us know.

Thanks

Your answer

KranthiPakala-MSFT 46,642 Reputation points Microsoft Employee Moderator

2023-02-01T23:22:39.95+00:00

Hi Adf2023,

As per the error description, seems like large number of Data Flow activity runs are occurring concurrently on the integration runtime.

Ref doc: Error code: 4502

Or may be hitting any of the below limitations. Azure Data Factory limits

Have you tried setting the pipeline concurrency to 1 and see if that helps to resolve the issue?

If the above checks don't help, I would recommend filing a support ticket for deeper investigation.

Do let us know how it goes.

Thanks
Adf2023 10 Reputation points

2023-02-02T14:55:44.41+00:00

@KranthiPakala-MSFT

Here is the config I have in IR which is dedicated to only this PL and Source (ADLS) not even have data to extract.

Yes, I set the pipeline concurrency to 1

Here is the overview of PL.

Export Data to ADLSGen2 by using Azure Synapse link push

DataVerse TO SQL ADF PL to load data from ADLS to Database(using Dataflow upsert method) I am wondering synapse link always actively pushing data to ADLS and when we run PL to extract data if that may cause issue. But i tried different trigger time and nothing resolved the issue.
KranthiPakala-MSFT 46,642 Reputation points Microsoft Employee Moderator

2023-02-03T22:35:03.0933333+00:00

Hi Adf2023,

Thanks for your response. If this issue persists consistently, then I recommend filing a support ticket for deeper investigation so that a support engineer can take a deeper look into this issue. Please do share the SR# once it is filed so that we can track it internally. In case if you don't have a support plan, please let us know.

Thanks

Share via

There are substantial concurrent MappingDataflow executions which is causing failures due to throttling under Integration Runtime 'IR(Manage Private Network)',

Your answer