ADF Pipeline error: Request content size is too large

Question

I have a pretty basic ADF pipeline that checks a subdirectory for a group of flat files and based on the name of the file, it will load data into a SQL staging table. The ForEach activity has 7 If Condition activities which will run based on the file name parameter that's passed. The pipeline completed successfully when I had 6 If Condition activities. I added a 7th that isn't doing anything different than the other 6, I'm just passing a different file name parameter. The file being loaded is only 10KB in size and similar in content to the other files that loaded successfully when the pipeline only contained 6 If Condition activities. When I try to debug the pipeline, I'm getting the following error: ErrorCode=RequestContentTooLarge, ErrorMessage=Request content size is too large

When I try to see where the failure is occurring, it's not showing me the failure details. Per the screenshot below, it says "Pipeline details could not be fetched from the server. The local pipeline configuration is shown." It looks like the first 2 activities complete but there's some kind of issue with the ForEach activity.

Answer 1

Hi @Matt Michalowski ,

Thanks for using Microsoft Q&A !!
You encounter this error when the pipeline with the included resource (i.e. datasets, linked service) passes the size limit allowed in a request payload. I believe when you are adding the 7th activity the payload size is getting past the defined limit.
The possible option is to break down the pipeline into child pipelines and execute them separately. Or have a parent pipeline execute multiple child pipelines.
Will it be possible for you to break your pipelines that exhibit this issue into multiple smaller pipeline?
You can use the Execute Pipeline activity that can help with this, please check this documentation for implementing the same.

Please let me know if this works for you or if you have any other questions.

Thanks
Saurabh