ADF: Write File names into file

Question

I want to write all filenames I got from my data source into a txt file where each filename stands in its own line.
I used a Get Metadata activity to get all childitems and a for loop iterates over the childitems. A Append variable appends the variable and after the for loop i use Set variable to save the array into a variable. After that I use the additional Columns in the Copy data activity. But the additional column doesn't accept an array variable so I have to convert it to a String. Then the output file is a txt file where everything stands on one line and the square brackets from the array and quotationmarks are also printed.

I also tried this approach (first answer from KranthiPakala-MSFT) to put the filenames into a json file, but the output file always stays empty.

I also read about using data flow for that purpose? How would that work? Or should I stay with the Copy data method?

Answer 1

Hi @Calathea ,

Welcome to Microsoft Q&A platform and thanks for posting your question.
I understand your scenario, you are trying to copy name of files present in a folder as each row in the output txt file. Please let me know if my understanding is incorrect. Your approach is correct till the point you have done the pipeline designing.

1. Use GetMetadata activity to fetch the fileNames by selecting childItems in the FieldList.
2. Use ForEach activity to iterate through all the fileNames and inside ForEach, use Append variable to concatenate filenames as array variable.
3. After the ForEach activity, use Copy activity having source dataset pointing to a dummy file having a single column , add an additional column Name=@{variables('fileNameJson')} . In the mapping tab, remove the column coming from dummy file and only keep additional column.
4. Post that, call dataFlow activity which has the following transformations:
5. Use Source transformation and create a new dataset pointing to the output .txt file generated from the copy data activity, make sure to select 'No delimiter' in the dataset.
6. Use derived Column transformation and select 'Column_1' and provide this expression split(replace(replace(replace(Column_1, '[', ''),']',''),'"',''),',') to convert string into array.
7. Use flatten transformation and unroll by Column_1[]
8. Use sink transformation and in settings, select output to single file in 'FileName option' and provide the fileName in the textbox and set single partition .

Here are the gif for reference:

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you.
  Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators