Data Flow using Temp File on SFTP by Default

Question

I am creating a process that picks up a txt file based on "blob created" event trigger in ADF Data Flow. Once this txt file triggers, my data flow picks up the file translates and maps it to a CSV. On my sink I have it connected to an SFTP connection that is sending the file to the same blob storage container that the txt source comes in. When a CSV enters this folder there is another trigger for "Blob Created" event listening for a CSV which it then moves into another process.

The issue I am having is the CSV trigger does not seem to work because data flow by default outputs to SFTP with temp files. In Copy Data flow I have this working fine as you can unselect send temporary files but I do not see this option in the Sink of Data Flow. Can someone please help me?

Answer 1

There is currently no option to disable temporary files in the Sink of Data Flow.

You can use a Copy Data activity instead of a Data Flow activity**.** As you mentioned, you can disable temporary files in the Copy Data activity.

Or you can use a custom script activity to move the temporary file to the target location with the ADF Bash Script or ADF PowerShell script activities to perform this task.

#!/bin/bash

# Get the path to the temporary file
temp_file_path=$1

# Get the path to the target location
target_location=$2

# Move the temporary file to the target location
mv $temp_file_path $target_location