How to Increase Web page response in Web activity of the Azure data Factory?

Question

Hello Team,

We are using the web activity in azure data factory. The maximum web page response we can receive it 4 MB and if it is exceeding this the pipeline is getting failed.
So, we are doing the web pagination and adjusting the count to get the web page response as 4 MB.
For example :- let's say we are having total 100 records (20 MB total Data) in one of the rest API table and problem is all the records can't be fetched as it will exceed the 4MB limit.
So by doing pagination we are taking 25 records (4 MB Data) at a time and then again the next 25 records and so on.
But it is very time consuming as the data volume is very huge.

So is there any possibility we can increase the web page response more than 4MB , so that we can get the more number of records.
and if the above scenario is possible, can you please share the solution/document for the same.

Thank you.

Answer 1

Hello @Dheeraj ,

Thanks for the ask and also for using the Microsoft Q&A forum.

The maximum supported output response payload size is 4 MB.

Possible workaround is to restrict the size of the output or use pagination in the response, but it is very time consuming as you said.

But feel free to submit your feedback to ADF product team through ADF user voice forum. All the feedback shared in this forum are reviewed and monitored by ADF engineering team.

ADF feedback forum: https://feedback.azure.com/forums/270578-data-factory

Hope this helps. Do let us know if you any further queries.