Azure Data Factory error creating blob

Question

Hi,

I have a pipeline in Azure Data Factory that queries Cosmos DB for some data and writes this data in a blob storage. It was working until it wasn't. I changed the query in the copy activity for a specific scenario, and when I ran it again it failed with an error at the creation of the blob. Note it doesn't complain about the query I changed, but about the creation of the blob. I also have tested the query and it's valid. Here's the error it presents:

ErrorCode=UserErrorFailedFileOperation,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The file operation is failed, upload file failed at path: 'processed-data-restore/extract/auto'.,Source=Microsoft.DataTransfer.Common,''Type=Microsoft.Azure.Documents.BadRequestException,Message=Message: ActivityId: 8599f89e-037f-4d95-a509-319d2a4b8625, documentdb-dotnet-sdk/2.5.1 Host/64-bit MicrosoftWindowsNT/6.2.9200.0,Source=Microsoft.Azure.Documents.Client,''Type=System.NotImplementedException,Message=The method or operation is not implemented.,Source=,'

Since the activity is just a configuration, not code, I'm out of ideas as how to make it work.

Regards

Answer 1

Hi Joao Borba,

Thanks for reaching out to Microsoft Q&A.

Based on the error message, it appears that the issue lies with the file operation during the blob creation step rather than the query itself. The error message suggests that the problem occurred during the file upload process to Azure Blob Storage. It suggests that the pipeline is failing when trying to create the blob file at the specified path: 'processed-data-restore/extract/auto'. The error message also mentions a BadRequestException from the Cosmos DB .NET SDK.

Some potential causes and suggested fix:

• Ensure the blob storage container and path exist and that the data factory has the necessary permissions to write to that location. Also, the network to blob should be stable.
• Check the cosmosdb query being used in the copy activity. Although the error doesn't mention the query, it's worth verifying that it's still valid and returns data as expected.
• If you recently changed the cosmosdb query, try reverting the change and see if the pipeline runs successfully again.
• Increase the number of retries and the retry interval in the copy activity settings in case the issue is temporary.
• Verify if there are any service limitations or throttling issues on the Cosmos DB or Blob Storage side
• Enable detailed logging in the copy activity to get more information about the failure.
• If the issue persists, consider splitting the copy activity into two separate activities: one to read from cosmos db and another to write to blob storage. This can help isolate the problem.
• As a last resort, try using a different method to copy the data, such as the azure cosmos db mongo api connector instead of the sql api.

Sometimes, the issue may be transient. Retrying the pipeline can help confirm if it's a temporary problem.

Please 'Upvote'(Thumbs-up) and 'Accept' as an answer if the reply was helpful. This will benefit other community members who face the same issue.