Copy Data activity Fails to fetch latest file from ADLS

Question

Hi All,

I am using ADF to get Latest file from ADLS but my copy data activity is failing as its need something the wildcard path either a file name or *.

This is quite confusing I can predict which file would be the latest and I don't want all the file to get copied.

Error - ErrorCode=FormatBasedDatasetMissingFileName,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=One of the property "file name" or "wildcard file name" is required. If you want to copy all the files from a folder instead of copy single file, you can put "*" in your wildcard file name,Source=Microsoft.DataTransfer.ClientLibrary,'

Answer 1

@Sarvesh Pandey

Thanks for reaching out to Microsoft Q&A.

I understand that you’re trying to use Azure Data Factory (ADF) to copy the latest file from Azure Data Lake Storage (ADLS), but you’re encountering an error because the copy data activity requires a file name or a wildcard path. Here’s a possible solution:

You can create a pipeline in ADF to get the latest file from a folder in ADLS.his involves using the GetMetadata and If Condition activities within a ForEach loop.

Here’s a step-by-step guide:

You could set modifiedDatetimeStart and modifiedDatetimeEnd to filter the files in the folder when you use ADLS connector in copy activity.

Maybe it has two situations:

1.The data was pushed by external source in the schedule,you are suppose to know the schedule time to configure.

2.The frequency is random,then maybe you have to log the pushing data time in another residence,then pass the time as parameter into copy activity pipeline before you execute it.

---

I try to provide a flow for you in ADF pipelines as below:

My sample files in same folder:

Step1,create two variables, maxtime and filename:

maxtime is the critical datetime of specific date, filename is empty string.

Step2, use GetMetadata Activity and ForEach Activity to get the files under folder.

GetMetadata 1 configuration:

ForEach Activity configuration:

Step3: Inside ForEach Activity,use GetMetadata and If-Condition, the structure as below:

GetMetadata 2 configuration:

If-Condition Activity configuration:

Step4: Inside If-Condition True branch,use Set Variable Activity:

Set variable1 configuration:

Set variable2 configuration:

All of above steps aim to finding the latest fileName, the variable fileName is exactly target.

---

Addition for another new dataset in GetMetadata 2

Method 2: Using Azure Functions

Method 2: Using Azure Functions

1. Develop an Azure Function:

• Create an Azure Function triggered by a blob change event in your ADLS folder.
• Inside the function, access the blob metadata (including "LastModified") to identify the latest file.
• Use the Azure Data Factory integration features within the function to trigger a Copy Data activity specifically for the latest file.

Hope this helps. Do let us know if you any further queries.

---

If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.