How to copy files based on the last modification' if file is not availble then send an email alert

Question

Hi Community,

I had question is
I have a SFTP(source) server which is run every day.

I have an SFTP (source) server that runs every day.

I'm receiving files from SFTP to ADLS (if file today is file 1 _24_04_2022.csv) etc .... I just want to copy the file that has the date today, if that file is the only one we need to copy today, other wise don't copy please send an email

Please help me

Answer 1

Hi @HemanthKumar loka ,
Thankyou for using Microsoft Q&A platform and posting your query. As I understand your query, you want to fetch the data from a file with today's date and copy the same to your destination datastore using ADF pipeline. Please let me know if my understanding is incorrect.

These are the steps you need to perform to achieve that:

1. Use GetMetadata activity pointing the dataset to the folder and using childItems in the field list to get the list of fileNames in that folder.

2. Use ForEach activity to iterate through each of the fileNames and pass this value as Item: @activity('Get Metadata1').output.childItems

3. Use If block inside ForEach with the following expression:
@equals(split(substring(item().name,add(indexof(item().name,'_'),1),add(lastIndexOf(item().name,'.'),-2)),'.')[0],formatdatetime(utcnow(),'dd_MM_yyyy'))

3. In the true condition of If block, add copy activity and use wildcard file path in the source dataset and provide fileName as @item().name

As you can see in the output , out of all the files present in the folder, only file with today's date got copied .

Hope this will help. Please let us know if any further queries.

------------------------------

• Please don't forget to click on or upvote button whenever the information provided helps you.
  Original posters help the community find answers faster by identifying the correct answer. Here is how
• Want a reminder to come back and check responses? Here is how to subscribe to a notification
• If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

Answer 2

Hey,
You can use the below steps for your query:

1. use get meta data activity to get the list of all files within the folder
2. use if activity to check whether there is any file with todays date

somewhat similar :
https://learn.microsoft.com/en-us/answers/questions/136815/lates-file-extraction-azure-data-factory.html

If yes, then use copy activity else use web activity to trigger logic app for email notification.
learn.microsoft.com/en-us/azure/data-factory/how-to-send-email