How to Increase Web page response in Web activity of the Azure data Factory?

Dheeraj 6 Reputation points
2020-12-17T09:35:15.127+00:00

Hello Team,

We are using the web activity in azure data factory. The maximum web page response we can receive it 4 MB and if it is exceeding this the pipeline is getting failed.
So, we are doing the web pagination and adjusting the count to get the web page response as 4 MB.
For example :- let's say we are having total 100 records (20 MB total Data) in one of the rest API table and problem is all the records can't be fetched as it will exceed the 4MB limit.
So by doing pagination we are taking 25 records (4 MB Data) at a time and then again the next 25 records and so on.
But it is very time consuming as the data volume is very huge.

So is there any possibility we can increase the web page response more than 4MB , so that we can get the more number of records.
and if the above scenario is possible, can you please share the solution/document for the same.

Thank you.

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
9,509 questions
{count} vote

1 answer

Sort by: Most helpful
  1. SaurabhRoy-MSFT 156 Reputation points Microsoft Employee
    2020-12-18T08:31:52.97+00:00

    Hello @Dheeraj ,

    Thanks for the ask and also for using the Microsoft Q&A forum.

    The maximum supported output response payload size is 4 MB.

    Possible workaround is to restrict the size of the output or use pagination in the response, but it is very time consuming as you said.

    But feel free to submit your feedback to ADF product team through ADF user voice forum. All the feedback shared in this forum are reviewed and monitored by ADF engineering team.

    ADF feedback forum: https://feedback.azure.com/forums/270578-data-factory

    Hope this helps. Do let us know if you any further queries.