Azure Data Factory Data Set - the most recent file

Question

Hi,

To validate incoming data to Azure Data Lake, I have built Pipe and Data flow (data flow is run inside the pipe). When a new file is added to the folder, it should immediately activate and use the newest file from this folder (so that it will be the added file). Two things are missing.

1. Event based trigger
2. I have to define the dataset, so the most recent file is always taken from the folder. Both in Pipe and Data Flow, the newest file needs to be used. Right now, I have no idea how to do that.

My pipe begins this way. And I need to use the newest file in the "Get Metadata Lease" activity here. The file used now is hardcoded,

The beginning of my data flow is here. The most recent file should be used here as well.

Data file used in the solution is here. So I need to change "Lease.csv" and make ADF to take the most recent file

Greetings
Andrzej

Answer 1

Hello @Gabryel, Andrzej and welcome back to Microsoft Q&A.

If I understand correctly, what you really want to know is, "How do I get the file name from an event trigger?"

For documentation, see step 10 and step 11 in how-to-create-event-trigger.

The idea, is to pass the filename and/or folderpath into pipeline parameters from the event trigger. Then you pass these pipeline parameters into your datasets through the activities.

So, step 1: create some pipeline parameters
Step 2: From your pipeline editing screen, do either new trigger or edit trigger. It is important you do it from the pipeline editing screen and not the manage > triggers screen because we want to use the context of your pipeline to set the parameters. In the last stage of creating/updating the trigger you can specify the value of parameters. We want to use:

@triggerBody().folderPath  
@triggerBody().fileName  

Step 3: we need a parameterized dataset


Step 4: pass the pipeline variable into the activity to be used in the dataset parameters