Data Factory Copy Data function dropping records

Question

I have set up a copy data pipeline in Azure DataFactory. The table I'm copying from contains 1500 records. The copy data step only read 872 records. I've tried to find records in the input table that seem to have bad data but nothing seems to be excessive long text or the wrong data type. I also looked for duplicate keys but I don't find any fields in the input record layout that indicate they are a key. Does anyone have suggestions as to what I should look for or try to change to resolve the issue?

Answer 1

I'm having the same issue.

My source is a parquet file and my sink is a stored procedure with user defined table type.

I have fault tolerance and logging enabled, when I run the pipeline in debug the copy data activity reports:

"rowsRead": 53303,  
"rowsCopied": 53303,  
"rowsSkipped": 0  

I have a simple truncate and insert (from the UDTT) in the procedure to debug the issue but I'm only seeing 4953 rows returned from the UDTT, it looks like they are the last 4953 in the file.

I have tried altering the data types in the UDTT but no difference, if I set up an alternat copy data activity to insert the data as a new table I get the complete file.

We are using SQL procedures as a sink as the processing of the file (bulk\Net etc) is based on other parameters.

Any help would be appreciated.

Answer 2

No I didn't enable skip incompatible rows. The problem seems completely random. Each time I run the Copy Data step I can't be sure what the result will be. Sometimes it copies all of the records and if I run it again a minute later it doesn't copy all of the records. There isn't any error message. Does anyone have a suggestion?

Answer 3

Hello @Mark Eckert and welcome to Microsoft Q&A.

So it sounds like you are missing expected rows. In the Copy Activity settings, did you be chance enable "Skip Incompatible rows" ? See below screenshot.

The Logging options allow the incompatible rows to be written to a file of your choice.