Copy data to blob storage fails with UserErrorFailedFileOperation,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Upload file failed at path

Anonymous

When copying data from cosmos DB to Blob storage I get an error:
"UserErrorFailedFileOperation,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Upload file failed at path orchestrationsarch/20220406temp\20220406bigFile.json.,Source=Microsoft.DataTransfer.Common,''Type=Microsoft.Azure.Documents.RequestTimeoutException,Message=Request timed out."

This is my sink configuration:

The action fails just after a short time which is odd because the result set of the query is not that big. Also, I don't know why there is a backslash in the path: "orchestrationsarch/20220406temp\20220406bigFile.json"

Can anyone tell me what could be wrong?

PRADEEPCHEEKATLA-MSFT 85,746 Reputation points Microsoft Employee

2022-09-02T07:00:29.717+00:00

Hello @Anonymous ,

Can you please confirm if you are able to preview data on the path which you have selected in the file path?

And also does it work when you provide full path as hard coded?
PRADEEPCHEEKATLA-MSFT 85,746 Reputation points Microsoft Employee

2022-09-08T08:36:57.41+00:00

Hello @Anonymous ,

Follow up to see if you have had a chance to see the previous response. We need the following information to understand/investigate this issue further.
Anonymous

2022-09-19T10:32:08.97+00:00

Please take a look at my "answer", by mistake I replied there.

3 answers

Anonymous

2022-09-19T08:33:57.857+00:00

The path doesn't really matter, slash and backslash nither. I'm able to access any other file in the container that was created by this pipeline. I just don't know why the outcome of this pipeline is so random. Sometimes it passes with no problem, sometimes it just waits from 1 minute to several hours.

A successful run for a large amount of data:

A failed run for no particular reason:

The error message is misleading, I don't know if this is a problem on the CosmosDB or the Blob side. I added logging to the copy activity, but it doesn't show anything useful. Is there a place where I can find detailed logs from the pipeline runs?
Please sign in to rate this answer.

1 person found this answer helpful.
PRADEEPCHEEKATLA-MSFT 85,746 Reputation points Microsoft Employee

2022-10-12T10:26:45.143+00:00

Hello @Anonymous ,

Apologize for the delay in response.

Did you resolve the issue or still experiencing the error message?

Anonymous

2022-10-21T10:24:33.787+00:00

The problem is still there.

PRADEEPCHEEKATLA-MSFT 85,746 Reputation points Microsoft Employee

2022-10-25T04:59:15.093+00:00

Hello @Anonymous ,

This issue looks strange. For a deeper investigation and immediate assistance on this issue, if you have a support plan you may file a support ticket.
Sign in to comment
DevWeb 26 Reputation points

2022-10-31T15:27:14.177+00:00

Do we have a solution for this issue? I am experiencing the same problem right now. The source storage container has many folders, it appeared that the one folder that has more nested folders (4 level) was choking here with the error message and the rest of the folder paths got copied without a problem.
Please sign in to rate this answer.

1 person found this answer helpful.
Anonymous

2022-11-02T10:11:41.427+00:00

I've created a support ticket to escalate this issue. Several MS departments are working on this because they have never encountered such a problem :P If I have any news I will post them here.

DevWeb 26 Reputation points

2022-11-11T23:16:46.673+00:00

Seems like a Microsoft bug to me. In my case, the File Path was broken down into 2 parts where you have 3, e.g. dataset().container / dataset().destination. I also have the '\' separator in the Directory part. But I got the same error, and it always failed at a specific location inside the specified Container/Directory structure, yet other containers or other directories under the same container has no issue. I have opened a ticket with Microsoft since 11/1/2022 and so far there has been no progress.

Anonymous

2022-11-14T07:54:38.937+00:00

I changed the sink to another CosmosDB container and the result was almost identical:
'Type=Microsoft.Azure.Documents.RequestTimeoutException,Message=Request timed out.
ActivityId: 19318520-e983-47be-b2fb-663ef123142e, Request URI: /dbs/BUcpAA==/colls/BUcpAKjiORc=/docs, RequestStats: , SDK: documentdb-dotnet-sdk/2.5.1 Host/64-bit MicrosoftWindowsNT/6.2.9200.0,Source=Microsoft.Azure.Documents.Client,''Type=System.Threading.Tasks.TaskCanceledException,Message=A task was canceled.,Source=mscorlib,'

I think it proves that the Blob storage and the destination path are not the problems.

I was testing my pipeline with different amounts of data and found out that anything above 300-400 MB would crush it. Then I changed my pipeline so it would run for every hour of the day instead of the whole day and it worked. I suggest you do the same and maybe it will also work for you.

DevWeb 26 Reputation points

2022-11-15T14:55:06.697+00:00

The problematic Blob storage path does not contain a large amount of data in my case. It does have many nested folder structures (6 levels) but a small number of small-sized files. I will have to figure out a workaround for this problem though. Thanks for the suggestion.

James Baxter 0 Reputation points

2023-01-19T00:30:08.01+00:00

I was getting the same error, I eventually figured it out when using the 'preview data' button surfaced the underlying error. Turns out my postgres table had an unusual datatype from postgis (geometry) which the underlying library used in the pipeline did not recognise. I solved it by changing my copy operation from targeting a table to being a query, and explicitly casting the geometry column using ::TEXT.

Probably not going to help you with cosmos though :|

DevWeb 26 Reputation points

2023-01-19T01:18:14.4333333+00:00

In my case, it was a Microsoft bug as they are trying to cope with blob storage with metadata and if you have metadata specified as null (the API supports it but the portal site does not), the error is bound to occur in ADF copy activity.

Tian, Xinyong 0 Reputation points

2023-01-20T16:15:53.07+00:00

I have similar error message, but Failure happened on 'Source' side. The source is query from serverless sql pool's external delta tables. My walkaround is to set retry to 5 (default is 0) in General tab of copy activity. Sometimes it succeeded after 4 trials/several hours.
Sign in to comment
Lokesh Rajpoot 36 Reputation points

2023-04-19T11:20:03.9366667+00:00

Hi @Anonymous i was also facing this same issue and that is resolved now the issue was on the source data there are some data that were not satisfying the condition. I was applying date casting and there were some data 202300501 that raised error in casting as DateTime after handling this my pipeline is fixed now Thanks
Please sign in to rate this answer.

1 person found this answer helpful.
Gulhasan.Siddiquee 101 Reputation points

2023-05-03T08:15:17.64+00:00

Hi @Lokesh Rajpoot I am also facing the same issue . But I have to choose only those files which are modified 2 months ago or earlier .So I have to apply Date casting .

Did you remove the Date Casting ?How did you handled it ? Can u please let me know.

here is Screenshot of my pipeline.

Lokesh Rajpoot 36 Reputation points

2023-05-11T13:07:06.8+00:00

you can cast this out of the foreach activity. If your watermark table is on the source that supports sql query you can easily do this in lookup activity as well.
Sign in to comment

Share via

Copy data to blob storage fails with UserErrorFailedFileOperation,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Upload file failed at path

3 answers