ADF Copy Data activity issue

Question

ADF Copy Data activity issue

Chi, Dave 61

Hello,

We have an ADF pipeline that is triggered daily for moving data from the on-prem server to the data lake storage. The data to be moved is incremental data from several tables. The pipeline runs usually go well, but on some days we run into an error. If we rerun the pipeline from failed activity then the run will succeed. This, however, involves manual action. Especially on the weekends, we don't want this to happen.

The error occurs in the Copy data activity. Within this activity, the data from an on-prem SQL Server table will be moved to the Data Lake Storage in a parquet file. We are using a self-hosted integration runtime.

We are getting different error messages:

Would you be able to help us out on this issue?

Answer accepted by question author

0 additional answers

Your answer

Answer 1

Vaibhav Chaudhari 38,971 Volunteer Moderator

It's showing memory related errors so that it something you can look into it.

Troubleshooting guide for Self hosted IR is here - https://learn.microsoft.com/en-us/azure/data-factory/self-hosted-integration-runtime-troubleshoot-guide

And you can set up the retry on Copy activity to rerun when it fails

----------

Please don't forget to Accept Answer and Up-vote if the response helped -- Vaibhav

Saurabh Sharma 23,866 Reputation points Microsoft Employee Moderator

2021-08-10T20:36:47.163+00:00

Hi @Chi, Dave ,

We haven't heard back from you. Did anonymous userchaudhari's response helped you or do you still require help on this issue ?

Thanks
Saurabh
Chi, Dave 61 Reputation points

2021-08-11T07:46:11.79+00:00

Hi anonymous userChaudhari , @Saurabh Sharma ,

Thank you for the retry suggestion. This will help when the activity fails.

Regarding the memory issue, it is not clear to me how to exactly resolve this. The provided documentation doesn't give an elaborate explanation.
Saurabh Sharma 23,866 Reputation points Microsoft Employee Moderator

2021-08-19T19:56:01.053+00:00

anonymous userchaudhari Any suggestions on this ?

Thanks
Saurabh
Vaibhav Chaudhari 38,971 Reputation points Volunteer Moderator

2021-08-20T05:33:59.733+00:00

@Chi, Dave - Try creating a separate new SHIR to see if that works smooth

To fix the existing issue, probably you can raise a support ticket from Azure portal so that support team can work closely and take a look at logs to fix the issue
Chi, Dave 61 Reputation points

2021-08-26T10:23:12.74+00:00

Hi anonymous userChaudhari ,

Thanks for the suggestion. I wil raise a new support ticket for the issue.

Share via

ADF Copy Data activity issue

0 additional answers

Your answer