Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
7,180 questions
Source HTTP response is greater than 4 MB for copy activity
ssalun
26
Reputation points
Hi ,
I am working on a pipeline which contains copy activity to copy data from HTTP response to Azure SQL database.
I am getting the response and the activity is completed successfully but no rows written to the SQL server table.
After doing google search came to know about the limitations for HTTP payload which is 4 MB. The work around is batching or paging the response, I would like to seek your help on batching the copy activity. If anyone has faced similar issue could you please share your knowledge?
Thanks.
{count} votes
-
HimanshuSinha-msft 19,196 Reputation points Microsoft Employee
2022-04-21T21:52:29.47+00:00 Hello @ssalun ,
Thanks for the question and using MS Q&A platform.
I undertand the ask , but are you sure that you are hitting the lmit . If you hit the limit you should get the below error .
The length of execution output is over limit (around 4MB currently)
I am bit confused as you toldI am getting the response and the activity is completed successfully but no rows written to the SQL server table.
We will wait to hear from you . .
Thanks
Himanshu -
ssalun 26 Reputation points
2022-04-22T14:29:30.817+00:00 Hi Himanshu,
You are right even i was expecting failure.I am not sure if i am missing one something else.
The response i am getting from http is around 12 MB and it executes the copy activity successfully but i see no records in the Azure sql table. I saw the sink connection looks good.
Sign in to comment