Data Factory Not Writing To an Empty CSV FIle

Question

I have a data flow that is very simple that I am trying am importing data from an azure data lake gen2 blob container renaming columns, filtering, and exporting it to a different named csv file in a different folder that is empty except for the headers which is what was in there to start. It is this file that is to be overwritten with a file with data. Why it wont create the file from scratch is beyond me but this is my first data flow and its not working. I am getting to other files created though. One called success_ and another some random file called partial with some random numbers and letters after it with only three records in it. When I preview the sink node in debug i see all the data. What do I need to do to get the data to export? Is there a way to get this file to be created from scratch also?

Answer 1

Hi Chris t hanks for providing screenshot.Once you do this setting, you can give a name under file name portion in sink , for example test.csv.

My understanding is, you are trying to write to a specific filename and everytime you expect the file to be overwritten. Please correct if my understanding is not right.

Under sink , you see an option for file name. If that's chosen as default, ADF will create a file with random name everytime you run the data flow. On the other hand, if you set it to Output to single file, you will be able to write to a specific file. But for this, you also need to choose 'set single partition' in optimize tab of sink.

This will make sure to override the same file everytime you run.

Please let us know if you have questions.

Thanks