How to copy files based on the last modification' if file is not availble then send an email alert

HemanthKumar loka 226 Reputation points
2022-04-24T13:19:56.96+00:00

Hi Community,

I had question is
I have a SFTP(source) server which is run every day.

I have an SFTP (source) server that runs every day.

I'm receiving files from SFTP to ADLS (if file today is file 1 _24_04_2022.csv) etc .... I just want to copy the file that has the date today, if that file is the only one we need to copy today, other wise don't copy please send an email

Please help me

Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
10,084 questions
0 comments No comments
{count} votes

Accepted answer
  1. AnnuKumari-MSFT 31,996 Reputation points Microsoft Employee
    2022-04-27T13:54:07.38+00:00

    Hi @HemanthKumar loka ,
    Thankyou for using Microsoft Q&A platform and posting your query. As I understand your query, you want to fetch the data from a file with today's date and copy the same to your destination datastore using ADF pipeline. Please let me know if my understanding is incorrect.

    These are the steps you need to perform to achieve that:

    1. Use GetMetadata activity pointing the dataset to the folder and using childItems in the field list to get the list of fileNames in that folder.

    197005-image.png

    2. Use ForEach activity to iterate through each of the fileNames and pass this value as Item: @activity('Get Metadata1').output.childItems

    3. Use If block inside ForEach with the following expression:
    @equals(split(substring(item().name,add(indexof(item().name,'_'),1),add(lastIndexOf(item().name,'.'),-2)),'.')[0],formatdatetime(utcnow(),'dd_MM_yyyy'))

    197064-image.png

    3. In the true condition of If block, add copy activity and use wildcard file path in the source dataset and provide fileName as @item().name

    196986-image.png

    As you can see in the output , out of all the files present in the folder, only file with today's date got copied .

    196948-image.png

    Hope this will help. Please let us know if any further queries.

    ------------------------------

    • Please don't forget to click on 130616-image.png or upvote 130671-image.png button whenever the information provided helps you.
      Original posters help the community find answers faster by identifying the correct answer. Here is how
    • Want a reminder to come back and check responses? Here is how to subscribe to a notification
    • If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators
    0 comments No comments

1 additional answer

Sort by: Most helpful
  1. Nandan Hegde 30,951 Reputation points MVP
    2022-04-25T04:45:48.637+00:00

    Hey,
    You can use the below steps for your query:

    1. use get meta data activity to get the list of all files within the folder
    2. use if activity to check whether there is any file with todays date

    somewhat similar :
    https://learn.microsoft.com/en-us/answers/questions/136815/lates-file-extraction-azure-data-factory.html

    If yes, then use copy activity else use web activity to trigger logic app for email notification.
    learn.microsoft.com/en-us/azure/data-factory/how-to-send-email