This browser is no longer supported.
Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(144, 7.000000000000001%, 23%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EHL%3C/text%3E%3C/svg%3E)
Hiw to get new files from sftp to adls with copy activity and i won't be copy same file .
For example :-
I have a file nani in sftp and I copied file into adls by using adf.
Get meta data 1+ for each[get meta data+if conditions +set variable]+copy activity
Next day same file available but no modifications in the file , if try to don’t copy same file into adls .
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(240, 12%, 33%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ESA%3C/text%3E%3C/svg%3E)
Hi @HemanthKumar loka ,please go through below video , it talks about similar scenario. Thanks
Thank you @Samy Abdul for helping here.
Hi @HemanthKumar loka , Thanks for posting query in Microsoft Q&A Platform. Below is the high level step by step process which you need to follow to process only new files and avoid already processed files.
Step1: Have a log table in SQL to store your processed file names.
Step2: At the end of your pipeline once you process any files log that file name in to above created log table.
Actual Pipeline implementation from here
Step1: Lookup activity to fetch file names from log table.
Step2: GetMeta data activity to get file names from your storage.
Step3: Filter activity to filter already processed files from GetMeta data activity output and take only unprocessed files.
Step4: Using ForEach activity loop through unprocessed files and copy them.
Step5: Once your copy activity runs any files immediately log that file name to log table.
The video link shared by @Samy Abdul explains similar implementation. Kindly check it once.
Hope this helps. Please let us know if any further queries.
-----------------
Please consider hitting Accept Answer button. Accepted answers help community as well.
@ShaikMaheer-MSFT thanks for you wrote the answer my question.
I would like to add one more point to my scenario
I don’t have any SQL
So can you help me on this
Hi @HemanthKumar loka - You can consider storing processed files names into some log file too. But in that case your file will get over rider every time. So always we need to make sure real all old data in logfile and union that data with new data and save it. Its going to be too much efforts. Having some database table for this purpose will be good idea.