My Copy Activity only downloads partially my file to blob storage

Cristian Rodriguez 0

Hello,

I started to receive only partial data using a copy activity in ADF.

Recently, I detected that my pipeline is only downloading 1,743,275 rows from an HTTP endpoint. I tried hardly to debug my pipeline and it is not changing the output. I do not have any skip rows rule in Fault tolerance.

When I go to the same URL using python I can download the entire file.

These are images from my pipeline:

User's image

I tried to sink into Blob storage or Azure SQL with the same results.

It stops in this number of rows:

User's image

Thanks.

ShaikMaheer-MSFT 38,321 Reputation points Microsoft Employee

2024-07-03T16:45:00.89+00:00

Hi Cristian Rodriguez,
Thank you for posting query in Microsoft Q&A Platform.

Here your API actually downloads file or gives JSON response back?
Cristian Rodriguez 0 Reputation points

2024-07-04T06:47:54.2866667+00:00

Hi,

Yes it downloads the file with fewer rows than expected.

That's basically the problem. I tried to set the dataset with multiple configuration for Quotation characters, as one of the fields have some quotation characters inside, i.e.

'"Aggregated Fee for ', ' Markup', ' Interchange', ' Scheme"',

This is my last configuration:

Thanks
ShaikMaheer-MSFT 38,321 Reputation points Microsoft Employee

2024-07-06T15:58:50.8766667+00:00

Hi Cristian Rodriguez,

As its downloads file directly to destination. Could you please try using binary dataset and see if that helps?
Cristian Rodriguez 0 Reputation points

2024-07-07T06:51:43.1533333+00:00

Hello,

Already tried that option and still getting the same results or the server in the source disconnected the connection.

I tried several time with Binary files but I also got incompleted files.

Is there any failure in ADF?

Share via

My Copy Activity only downloads partially my file to blob storage