Copy Activity goes into Internal Server 500 error for rest identity.

Jadhav, Karan 10

Hello,
I am using copy activity and selecting rest_identity for call data from different API. I made a main and micro pipeline. The main pipeline consists of calling the token which is being passed to several micro pipeline. Out of 5 micro-pipeline, one of them is failing.
issue

The one which is failing consists of some large data size(in MBs) in comparison to other API calls which is in (KBs). I have tried increasing the timeout from general and source side. Still it gives in the Internal server 500. Also, I tried running the same in the Azure Portal where we run, test and monitor the code.
azure_portal

Can you please help? Thanks!

PRADEEPCHEEKATLA-MSFT 89,281 Reputation points Microsoft Employee

2023-03-31T05:30:31.78+00:00

Hello @Jadhav, Karan ,

We received your feedback that the answer provided on the thread was not helpful.

Kindly let us know what we could have done better to improve the answer and make your engagement experience good. We are here to help you and strive to make your experience better and greatly value your feedback.

Our engineer has provided a detailed answer which has clear steps for which you are looking for.

If you wish, you may re-surveying/rating for the engagement you received on the thread. Your feedback is very important to us.

Looking forward to you reply. Much appreciate your feedback!

Regards,

PRADEEPCHEEKATLA-MSFT

1 answer

HimanshuSinha-msft 19,476 Reputation points Microsoft Employee

2023-02-21T20:16:08.2533333+00:00

Hello @Jadhav, Karan ,
Thanks for the question and using MS Q&A platform.

I am assuming that the activity which is failing has a response in MB's as compared to the others which is in KB's. I am assuming that you are using Web activity and with all the information which you shared, I want to point to the fact that Web activity has got a hard limit of 4MB as response payload size and may thats the reason that the activity is failing.

https://learn.microsoft.com/en-us/azure/data-factory/control-flow-web-activity

Thanks

Himanshu

Please accept as "Yes" if the answer provided is useful , so that you can help others in the community looking for remediation for similar issues.
Please sign in to rate this answer.
Jadhav, Karan 10 Reputation points

2023-03-15T10:03:33.58+00:00

Hi @HimanshuSinha-msft, Thanks for your reply. I am using copy activity to call the Rest api which is giving me the above error, whenever I preview the data or run the pipeline, it gives internal server 500. However, I did try the same thing using web activity which gives me 2108 error encoded with internal server 500, https://learn.microsoft.com/en-us/azure/data-factory/data-factory-troubleshoot-guide. I tested the same using postman and it works fine. Can you please help me how I can increase payload or anyother activity which will help me getting the data.

Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.

HimanshuSinha-msft 19,476 Reputation points Microsoft Employee

2023-03-30T18:49:07.5233333+00:00

Hello @Jadhav, Karan ,

When you make the call from Postman and get a response back. Can you please copy the response and safe into file and check the file size , if its more then 4 MB , you are hitting the limit and if its less then we have a different issue .

Thanks
Himanshu
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Copy Activity goes into Internal Server 500 error for rest identity.

1 answer

Your answer