Copy Activity intermittently copying first row only?

Mats Elmsater 1

We just noticed that some of our Copy Activities have been copying one row of data for a few days, although the JSON file source contains multiple rows.
When we went back and re-ran the pipeline today the Copy Activity copied all records as expected.

On the instances where only the first row was copied, we received no errors and Rows Read was the same as Rows Written (both 1)
Our source is a Data Lake JSON file with format: Set of Objects
Our sink is a Azure SQL Database table

Has anyone experienced anything similar?

svijay-MSFT 5,241 Reputation points Microsoft Employee

2021-11-18T13:32:41.087+00:00

Hello @Mats Elmsater ,

Thanks for the question and using MS Q&A platform. Is there any other pattern that you are able to find ? I am assuming you are re-running the pipeline with out any modification of the file or pipeline.
Mats Elmsater 1 Reputation point

2021-11-18T14:53:35.773+00:00

Correct, when I'm re-running the pipeline I'm making no adjustments, other than the fact that it is the only copy activity running, since all other files have already copied successfully.
We are running multiple pipelines in parallel, all using a copy activity with a Data Integration Unit of 32. I would hope that is not causing any issues, as I would expect a failure if the copy activity timed out du to insufficient DIU, and not just copying the first record.

I also confirmed that the file was created well before the pipeline executes.
Mats Elmsater 1 Reputation point

2021-11-19T18:53:15.76+00:00

Hi @svijay-MSFT ,
We are continuing to experience the same issue. On initial run the Copy Activity only reads and writes 1 row, but on subsequent run all rows are read. Looking at the pipeline activities, the inputs and outputs are all the same (except the rows copied) as far as I can tell.

Nothing is done to the file that is being copied and the file is created several hours prior to the Copy Activity starts.
The file itself is a .json file with a set of objects (Concatenated objects)

Any insight would be most helpful.
Thanks
svijay-MSFT 5,241 Reputation points Microsoft Employee

2021-11-23T07:32:33.087+00:00

Hello @Mats Elmsater ,

That's Strange. I had been researching and I was not able to find any related to our issue. Having said that, this will need a deeper investigation and immediate assistance, if you have a support plan I'd
request you to file a support ticket. In case if you don't have a support plan, I will enable a one-time free support request for your subscription. Please do let us know.
Mats Elmsater 1 Reputation point

2021-12-01T13:57:33.69+00:00

Hi @svijay-MSFT ,

For now I have solved it by creating an individual pipeline for these specific files where I have a two Copy Activities running back to back.
The first one copies only 1 row (which is the issue), but by adding a second Copy Activity immediately after it copies all rows correctly.

So for now I think I have found a solution, but I'm definitely curious to see why this happens and I'm sure you are as well.
I don't believe we have a support plan, so if you can create a one-time request that would be great.

Thanks.
svijay-MSFT 5,241 Reputation points Microsoft Employee

2021-12-07T14:57:59.777+00:00

Thank you and apologies for the delay. I'll check & work internally on the one-time request for you and get back to you on this shortly.
GNSAMPAIO 26 Reputation points

2022-06-23T06:22:29.457+00:00

Hello All,

I am facing the exact same issue.

Tried the work-around of @Mats Elmsater with two consecutive copies but with no use.

I am also reading from REST API with Json output, I am able to preview the data but copy command to datalake in parquet file only reads one row.

In my case, I managed to load the data only once.
Denisa Pasha 25 Reputation points

2023-06-29T11:49:42.77+00:00

There's a trick to this. You import schemas, then you put the name of the array in collection reference then you import schemas again then it works

3 answers

GNSAMPAIO 26 Reputation points

2022-06-23T06:59:37.617+00:00

I think I solved the issue.

On Mapping, default array structure was (in my case) $['DetailedRecords'][0][TYPE] --> [TYPE] being TYPE the column name.
[0] is taking only first index.

I am mapping
Collection reference > $['DetailedRecords']
I tried mapping [TYPE] --> [TYPE] and it is working now
Please sign in to rate this answer.

5 people found this answer helpful.
Marcus Maldonado 1 Reputation point

2022-07-15T13:53:19.477+00:00

Many thanks my friend.

Saved the day and some working hours. :)

Victor Ochieng 1 Reputation point

2023-06-14T08:17:11.4766667+00:00

Thank you so much for this hint! Works like a charm!!
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.
Denisa Pasha 25 Reputation points

2023-06-29T11:50:07.57+00:00

There's a trick to this. You import schemas, then you put the name of the array in collection reference then you import schemas again then it works
Please sign in to rate this answer.

5 people found this answer helpful.

0 comments No comments
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.
Souilem, Mahdi 1 Reputation point

2022-05-06T15:02:55.637+00:00

Hi, i want to know if this problem is solved,
i m facing the same issue, i m making two copy activities, one from a REST API and the second from a JSON file, both of them are returning only the first row in my sql database,
i tried to copy this in a JSON file instead of sql database and it worked perfectly, it returned all the rows,
i m wondering why in sql database it s returning just the first row,
anyone have an idea about this issue?
thanks
Please sign in to rate this answer.
Mats Elmsater 1 Reputation point

2022-05-06T15:18:16.323+00:00

Hi, I have not re-tested since I found a workaround (see my comment above about using two consecutive copy activities).
If you are experiencing something similar to what I saw, I would recommend to try and use two consecutive copy activities as for some reason the second copy activity copied all rows after the first one only copied the first row.

Pete Florenzano 0 Reputation points

2023-02-22T15:44:26.82+00:00

I'm running into the same exact issue and tried both a secondary 'Copy Data' task and removing the 0 index with no success.

Any information would be greatly appreciated!

Thank you
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Copy Activity intermittently copying first row only?

3 answers

Your answer