Temporary Folders(ex: 0c9f41ec-6866-425e-b7b3-ad18cc5655eb) are creating in SFTP that cannot be deleted, after Sink activity from Azure Dataflow.

Question

Temporary Folders(ex: 0c9f41ec-6866-425e-b7b3-ad18cc5655eb) are creating in SFTP that cannot be deleted, after Sink activity from Azure Dataflow.

Anonymous

Temporary Folders(ex: 0c9f41ec-6866-425e-b7b3-ad18cc5655eb) are creating in SFTP that cannot be deleted, after Sink activity from Azure Dataflow.

I'm using Dataflow to copy files from Blob to Csv through transformations, at the end is Sink activity where Sink is SFTP.

There is no option that I can avoid these unwanted folders(ex: 0c9f41ec-6866-425e-b7b3-ad18cc5655eb) created alongside this original csv files.

These cannot be deleted in SFTP as this is protected file.

QuantumCache 20,681 Reputation points Moderator

2023-07-13T20:27:36.88+00:00

Hello @C, Chandy(GLOBAL-V) Welcome to Q&A forum.

Yes, we have seen this scenario, and can be controlled in 2 places, Copy Activity and the SFTP User permissions!!! below is a workaround solution to use Pipeline instead of Dataflow!

Scenario1:

SFTP user READ, WRITE on SFTP location+ Copy Activity un-Check temp file upload.

Scenario2:
I had to give the Delete permission to the SFTP user account. So that the user can delete the temporary files after Copy Operation. and un-check the Option in the Copy Activity if at-all you want to skip using the temp file concept. As shown in the below image.

Scenario3:

SFTP user permissions: Read, Write only--->ADF Copy Activity Fails (With the Temp file option enabled)

Please don’t forget to Accept Answer , wherever the information provided helps you, this can be beneficial to other community members as well.
Anonymous

2023-07-13T20:50:57.8166667+00:00

I'm using Dataflow sink as the last and it doesn't have Upload with Temp file option and I have Delete option in SFTP as I can delete other SFTP file but not this temp file

Please read the question before answering Satish. After searching all option on internet only I came to this community.
QuantumCache 20,681 Reputation points Moderator

2023-07-13T20:56:43.76+00:00

Apologies, I have overlooked!

If possible , you may want to use the Pipeline route instead of Dataflow.

But for Dataflow you are right, We don't have that option, let me test and get back!
Anonymous

2023-07-14T11:00:45.1233333+00:00

Thanks, but really want to have this solution in Azure Dataflow
QuantumCache 20,681 Reputation points Moderator

2023-07-14T14:36:09.56+00:00

Hello @C, Chandy(GLOBAL-V) The dataflow is indeed creating the Temp folders

Could you please add screenshot of any sample output?

I have escalated this issue internally to my team, will keep you posted.
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
QuantumCache 20,681 Reputation points Moderator

2023-07-14T19:05:23.6833333+00:00

Hello @C, Chandy(GLOBAL-V) Thanks for the information,

We really appreciate that you reported this issue over this forum. I have a confirmation from the product team that this need to be fixed from dataflow side and we need you to send us an email to work with the product and investigate the issue much deeper. Even though the above workaround for using pipeline copy Activity still is an option, we will try to fix the issue with Dataflow.

Please send an email to azcommunity@microsoft.com with the below details, so that we can create a one-time-free support ticket for you to work closely on this matter.

Email Subject : Attn Satish Boddu

Email Body content:
Thread URL: <Link to this thread. >

Subscription ID: your azure subscription ID is required!

Please let me know once you have done the same.
QuantumCache 20,681 Reputation points Moderator

2023-07-17T04:45:32.02+00:00

Hello @C, Chandy(GLOBAL-V) Just checking if we are still connected on this discussion? Please send us an email with above details and we would be more than happy to help you in this matter!
QuantumCache 20,681 Reputation points Moderator

2023-07-18T23:25:54.2266667+00:00

Hello @C, Chandy(GLOBAL-V) Could you please send us an email as per previous request, so that we provide you one time free technical support in this matter!
Aditya Verma 20 Reputation points

2024-04-22T22:52:50.94+00:00
Hi,

I'm facing the same issue while using SFTP sink via a Dataflow. The dataflow is creating temp folders / files and in the end, it tries to rename the file but fails due to reason:

Spark job failed in one of the cluster nodes while writing data in one of the partitions to sink, with following error message: Failed to rename

We need to write to SFTP sink from the Dataflow only but can't find any such option as 'useTempFileRename' which can be set to 'False' (like in copy activity).
Please help us how to make our Dataflow write directly to SFTP sink instead of creating temp files