ADF Copy Activity

Question

Hi,

Will there be a case where copy activity doesnt copy whole source data into destiantion, but still it succeeds without error.

for eg I have 100 records in a csv file and that is copied to azure sql db, but it has written only 50 rows. But there was no failure or error.

How do we ensure the whole source data is written into target system.

Answer 1

Hi @Santhi Dhanuskodi

Welcome to Microsoft Q&A platform and thanks for posting your query here.

As per my understanding you are trying to copy data from source to SQL DB using copy activity within Azure data factory. Based on the information you provided, it seems like the copy activity is succeeding without any error messages, but only a portion of the data is being successfully copied.

It seems you have not enabled fault tolerance, then chance of file getting skipped is negligible. Can you confirm even the output shows the number of rows read and written are same?

Will there be a case where copy activity doesnt copy whole source data into destiantion, but still it succeeds without error.

However, it is possible for the copy activity to succeed without error but not copy the whole source data into the destination. This can happen due to a variety of reasons, such as network issues, data type mismatches, or data truncation.
Kindly try the following steps and see if it helps:

1. Increase the number of Data Integration Units (DIUs) used by the copy activity.
2. Reduce the parallel processing of files by reducing the DOP value. Instead of copying all the files in one go, you can try copying them in smaller batches. This can help reduce the resource requirements of the copy activity and improve the chances of a successful transfer.
3. Try enabling session logging which would help to track each and every activity performed by the copy activity and log it in intermediate file. You can scan the Copy activity session logs to see which activity actually copied the files, and when. With this approach, you can easily find the root cause and fix your configurations in ADF.

How do we ensure the whole source data is written into target system.

To ensure complete data transfer you can compare the number of rows read from the source and written to the destination using the built-in metrics and logging within ADF. Monitor the Rows Read and Rows Written counters in the copy activity output.

For better understanding, kindly watch out:

https://www.youtube.com/watch?v=xbL0UL78x-M

https://www.youtube.com/watch?v=vQl87DJRyeQ&t=194s

I hope it helps. please let me know if you have any further questions.