Limitation API 4MB of data API HTTP Activity

Question

I am facing issues regarding the size of the output body request when i am trying to get data from a SOAP API using the HTTP Activity.

Does anyone found a solution to fix this problem ? The only solution i have found is to iterate on a more granular object but it took too many time to used this approch.

The length of execution output is over limit (around 4MB currently)

Answer 1

Hi Régimbald, Marc-André ,

Welcome to Microsoft Q&A platform and thanks for posting your query.

As per my understanding you are trying to fetch data from SOAP API using http connector , however you are facing 4mb limitation . Are you using lookup activity or web activity?

In case you want to load the data>4mb into file, you can still use copy activity and use http connector(binary format) in the source and ADLS gen2/Blob storage (binary format) in sink dataset and load it to the Azure storage in case the API doesn't support pagination.

To consume the data in the pipeline , The workaround is to design a two-level pipeline where the outer pipeline iterates over an inner pipeline, which retrieves data that doesn't exceed the maximum rows or size.

Hope it helps. Kindly accept the answer by clicking on Accept answer button. Thankyou

Answer 2

Hi Régimbald, Marc-André:

Thanks for reaching out to Microsoft Q&A.

Unfortunately, it is the hard limit

The maximum supported output response payload size is 4 MB.

You can try paginating your api results. ADF connector can support pagination, but you can use it only when your API response is pagination enabled.

Adding up another answer for your reference.

[https://learn.microsoft.com/en-us/answers/questions/1460361/the-length-of-execution-output-is-over-limit-(arou](https://learn.microsoft.com/en-us/answers/questions/1460361/the-length-of-execution-output-is-over-limit-(arou)

Please 'Upvote'(Thumbs-up) and 'Accept' as answer if the reply was helpful. This will be benefitting other community members who face the same issue.

Answer 3

In facts, i've already try it and it've been working.The main reason is because my HTTP endpoint is not public. I am currently request an "LocalHost" EndPoint.