Options for making http request for file > 500 KB

Question

I have an API I am calling in a copy activity and the amount of data I would like to request each time is over 500 KB, which is documented as the max payload size for the http connecter. Other than making smaller batch calls, is there a way to retrieve more than 500KB?
https://learn.microsoft.com/en-us/azure/data-factory/connector-http#dataset-properties

Answer 1

Hello @DC-q489h2 ,

Thanks for the question and using Microsoft Q&A platform.

Unfortunately, there is no way to retrieve the HTTP request payload more than 500KB.

As mentioned in the document, if the payload size you want to pass to your web endpoint is larger than 500 KB, consider batching the payload in smaller chunks.

I would request you to provide the feedback here:

https://feedback.azure.com/forums/270578-data-factory

All of the feedback you share in these forums will be monitored and reviewed by the Microsoft engineering teams responsible for building Azure.

Hope this helps. Do let us know if you any further queries.

---------------------------------------------------------------------------

Please "Accept the answer" if the information helped you. This will help us and others in the community as well.