Copy Data activity fails when copying from ADLS to ADLS

Question

I'm currently attempting to use a Copy Data activity to copy a .json file over from what we treat as a "staging area" of our data over to our Datalake all within Azure Datalake Storage.

The files start out in .json format but are copied over into .avro format using a dataset.

Currently I am receiving the following error and am unsure what the exact problem is.

Operation on target Copy NE Machines File failed: ErrorCode=UserErrorFailedFileOperation,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The file operation is failed, upload file failed at path: 'datalake/data/2024/04/24'.,Source=mscorlib,''Type=System.ArgumentOutOfRangeException,Message=Specified argument was out of the range of valid values. Parameter name: size,Source=Microsoft.Hadoop.Avro,'

Answer 1

Hello Christopher Ackerman,

Greetings! Welcome to Microsoft Q&A Platform.

It looks like there was a failure during the file upload operation in your data transfer process. The error message indicates that there was a problem during a file operation in your data transfer process. Specifically, it occurred while trying to upload a file at the path 'datalake/data/2024/04/24'.

And this issue might also raise if the argument (parameter) was out of the valid range of values and error message= Specified argument was out of the range of valid values. Parameter name: size: The specific issue seems to be related to the size parameter. It suggests that the size value provided is not within the expected range.

To troubleshoot this issue, consider the following steps:

• Ensure that the specified file path 'datalake/data/2024/04/24' exists and is accessible. Verify that the path is correct and that the file you’re trying to upload is present.
• Examine the size of the file you’re attempting to upload. If it’s too large or exceeds any limits, it could trigger this error. Consider splitting large files or optimizing the data transfer process.
• If any data transformation or conversion is involved (e.g., from one format to another), verify that it’s correctly implemented. Incorrect data conversions can lead to unexpected errors.
• Ensure that the account or service performing the data transfer has the necessary permissions to write to the specified path in your data lake.
• Investigate if there are detailed logs available for the data transfer process. These logs might provide additional insights into the root cause of the issue.

refer - https://learn.microsoft.com/en-us/azure/data-factory/connector-azure-data-lake-storage?tabs=data-factory for detailed guidance on transferring data.

When working with Avro files, be aware of certain limitations:

Avro complex data types (such as records, enums, arrays, maps, unions, and fixed) are not supported in the Copy Activity. If your .json data contains any of these complex types, you may need to preprocess or transform it before copying to Avro format. refer - https://learn.microsoft.com/en-us/azure/data-factory/format-avro#copy-activity-properties for detailed guidance.

You can try retrying the file upload operation to see if the issue was a temporary one. If the issue persists, please do let us know for further traction.

Hope this answer helps! Please let us know if you have any further queries. I’m happy to assist you further.

---

Please "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.