Copy activity failed on heavy files

Question

Hello.

Today I faced issue when try to copy from gzip-ed csv file which ~24 MB archived and ~217 MB unzipped.
Copy activity just failed with the following error:

But I tried to do data preview and it works:

For small files it works fine.

Does anyone faced the same issue? Is there any not documented limitations for copy activity?

Thanks.

Answer 1

Hi @Алексей Грецов ,

Thanks much for using Microsoft Q&A forum for posting your query and sharing your findings.

Basically Azure data factory Copy activity performs source types to sink types mapping with the following flow:

• Convert from source native data types to interim data types used by Azure Data Factory and Synapse pipelines.
• Automatically convert interim data type as needed to match corresponding sink types, applicable for both default mapping and explicit mapping.
• Convert from interim data types to sink native data types.

Copy activity currently supports the following interim data types: Boolean, Byte, Byte array, Datetime, DatetimeOffset, Decimal, Double, GUID, Int16, Int32, Int64, SByte, Single, String, Timespan, UInt16, UInt32, and UInt64.

As you have used data types that aren't supported by ADF copy activity, it errored out.

Glad to know that you were able to figure it out and appreciate much for sharing it here as it can be beneficial to other community members reading this thread.

----------

• Please don't forget to click on and upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

Answer 2

Seems it is related to additional columns + dynamic mapping.
If I use imported schema - it works, but dynamic mapping doesn't work.

Answer 3

Yep, issue is related to mapping, especially data types.
In my case root cause was I used INT, BIGINT, DECIMAL(10,4) as sink, but after I replace it in the following way:
INT -> INT32
BIGINT -> INT64
DECIMAL(10,4) -> DECIMAL

It started to work.

Question closed