How to configure filepath to read in the latest file each time in azure data factory

Question

I have a new data set that is added daily into my azure datalake storage and i want to use this latest data in my pipeline. However, im not sure what to fill in at the 'File Name' column. I have a trigger that i set to run everytime the new file is added. My pipeline has Set variable activity that references the file name as well so leaving it blank causes this error:

My files are named like this: File_Date20230922.txt

Answer 1

Hey,

Since the ask is to get the file name that trigerred the ADF pipeline via blob trigger :

https://stackoverflow.com/questions/62052791/how-to-get-the-name-of-the-file-that-triggered-the-azure-data-factory-pipeline

Add a parameter to your pipeline, say, triggeringFile.

When you create the trigger, a form pops-out on the right side - after submitting the first page, a second page pops-out - this will ask for a value for the pipeline parameter triggeringFile. In that box, put @trigger().outputs.body.fileName

If the format you gave is the standard then your table name is just @{split(pipeline().parameters.triggeringFile,'_')[0]}

Answer 2

@ka Thanks for using Microsoft Q&A forum and posting your query.

I'm assuming you are using Storage event triggers to copy the files which are landed in your source folder on daily basis and you would want to get the file name dynamically added to your copy activity source configuration. If that is the case you can follow the steps as described in this document: Create a trigger that runs a pipeline in response to a storage event

You will have to create trigger parameter and add below system variables to retrieve the folder path and file name when file is landed in your storage folder.

Ref doc: System Variables -Storage event trigger scope

Hope this info helps.

---

Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.