ADF dataflow creating temporary folder

Jaganath Kumar 105

Hello everyone,

I'm currently generating a file using a data flow, and I've noticed that a temporary file is being created in a directory called _temporary/0/.

User's image

I have another pipeline that triggers upon the creation of the file mentioned above, and it moves the file to a third-party system. However, because of the temporary file, the trigger is reading the temporary file, attempting to process it, and then failing.

https://learn.microsoft.com/en-us/answers/questions/967167/adf-data-flow-temporary-0-directory

To make it clear, I have a same problem as above mentioned post.

Appreciate your help.

Thanks,

Jag

AnnuKumari-MSFT 31,061 Reputation points Microsoft Employee

2024-04-24T08:32:27.9233333+00:00

Hi Jaganath Kumar ,

Thankyou for using Microsoft Q&A platform and thanks for posting your query here.

I understand that you are facing an issue with a temporary file being created in a directory while generating a file using a data flow in Azure Data Factory. Due to this temporary file, the trigger is reading the temporary file, attempting to process it, and then failing.

Could you please share more details like since when are you facing this issue? Is it something that earlier the dataflow was not behaving this way but since sometime you are facing this issue?

Kindly share if any partitions are set along with the screenshot of dataflow configuration for partition. Thankyou
Jaganath Kumar 105 Reputation points

2024-04-25T04:21:20.73+00:00

Hi @AnnuKumari-MSFT ,

I appreciate your prompt response.

This is one of the technical requirement in the project. The pipeline should automatically trigger whenever files are generated in the blob storage and transfer them to the third party.

During my testing of the trigger, I encountered this issue. Here's the flow:

All the activity uses "Use current partition" and Sink uses 'Single Partition'

Hope this helps.

Thanks,

Jaganath
Jaganath Kumar 105 Reputation points

2024-04-25T23:56:40.5+00:00

Additionally log details from ADF:

Operation on target TransferFiletodev failed: ErrorCode=UserErrorFileNotFound,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=ADLS Gen2 operation failed for: Operation returned an invalid status code 'NotFound'. Account: 'devstoreadf'. FileSystem: 'container-dfcont'. Path: 'Load/devfiles/996e82ac-a4a5-4f8b-a9af-4ea758a43983/_temporary/0/_temporary/attempt_202404252352005686445654116841626_0000_m_000000_0/part-00000-9b6c090a-2419-446b-b769-60fe934984ff-c000.txt'. ErrorCode: 'PathNotFound'. Message: 'The specified path does not exist.'. RequestId: 'd29c5be7-801f-004a-676b-9749ab000000'. TimeStamp: 'Thu, 25 Apr 2024 23:52:51 GMT'.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=Microsoft.Azure.Storage.Data.Models.ErrorSchemaException,Message=Operation returned an invalid status code 'NotFound',Source=Microsoft.DataTransfer.ClientLibrary,'
Jaganath Kumar 105 Reputation points

2024-04-28T23:27:29.22+00:00

@AnnuKumari-MSFT

Just check if there is any solution regarding this issue.