My Azure data factory copy pipeline is not bringing over all of the data, how do I fix this?

Question

Hello,

I am having a data issue in Azure data factory. I am running an ingest copy function and in a specific table it is not transferring over 850,000 rows. I have rebuilt the pipeline and run through all of the troubleshooting steps when I have seen this issue in the past, however this time I am stuck.

Answer 1

Hi @v-maswin@microsoft.com ,
Welcome to Microsoft Q&A platform and thanks for posting your question here.
As I understand your query , while performing copy activity, few of the data is not getting copied to the sink , please let me know if that is not the ask here.

Kindly check if Fault tolerance option in copy activity settings is set to 'Skip incompatible rows' .

This option enables the copy activity to skip records which are errored out, instead of failing the activity. If it is enabled for you, that might be the reason of not getting entire records from source to sink.

For more details on fault tolerance, kindly check out: Fault tolerance in ADF
https://www.youtube.com/watch?v=BR9zlZnpPfU

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you.
  Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators