Source HTTP response is greater than 4 MB for copy activity

ssalun 26

Hi ,

I am working on a pipeline which contains copy activity to copy data from HTTP response to Azure SQL database.
I am getting the response and the activity is completed successfully but no rows written to the SQL server table.
After doing google search came to know about the limitations for HTTP payload which is 4 MB. The work around is batching or paging the response, I would like to seek your help on batching the copy activity. If anyone has faced similar issue could you please share your knowledge?

Thanks.

HimanshuSinha-msft 19,376 Reputation points Microsoft Employee

2022-04-21T21:52:29.47+00:00

Hello @ssalun ,
Thanks for the question and using MS Q&A platform.
I undertand the ask , but are you sure that you are hitting the lmit . If you hit the limit you should get the below error .
The length of execution output is over limit (around 4MB currently)
I am bit confused as you told

I am getting the response and the activity is completed successfully but no rows written to the SQL server table.

We will wait to hear from you . .
Thanks
Himanshu
ssalun 26 Reputation points

2022-04-22T14:29:30.817+00:00

Hi Himanshu,
You are right even i was expecting failure.I am not sure if i am missing one something else.
The response i am getting from http is around 12 MB and it executes the copy activity successfully but i see no records in the Azure sql table. I saw the sink connection looks good.