How to copy data from Network Share to landing when we have modified date before created date

Question

How to copy data from Network Share to landing when we have modified date before created date

Dwivedi, Upendra 0

Hi All,

We are encountering a challenge related to a known behavior of Microsoft file types such as TXT, CSV, Excel, and image files. When these files are copied and pasted, the Created Date gets updated while the Modified Date retains its original value.

In our data ingestion process, we compare the file's Modified Date with the last load date in the landing zone to identify new or updated files. Because the Modified Date remains unchanged after a copy-paste operation, some files are being missed during processing.

As a workaround, we considered using file metadata using get metadata activity and compare file modified date and created date with Last Load Date. However, due to a deeply nested folder structure and a large number of files, retrieving and processing metadata is significantly impacting performance.

Has anyone faced a similar scenario? If so, what approach or solution did you implement to reliably identify newly copied files while maintaining good performance?

Any suggestions or best practices would be greatly appreciated.

Thanks in advance.

0 comments

2 answers

Your answer

Answer 1

Hello Dwivedi, Upendra,

Welcome to the Microsoft Q&A and thank you for posting your questions here.

I understand that you are asking how you can copy data from Network Share to landing when we have modified date before created date.

You do not need to tune the LastModified filter, make ADF copy decisions from a manifest-backed ledger. By replacing the LastModified-only watermark with a manifest and ingestion ledger pattern. Generate a manifest from the source network share, store each file’s relative path, file name, size, created time, and modified time, compare it against a durable ledger of previously copied files, then copy only new or changed files. Update the ledger only after the copy succeeds.

After implementing the manifest-ledger approach, files copied into the network share will no longer be skipped simply because their Modified Date is older than the previous pipeline run. The pipeline will detect them as new by file path/key and copy them safely into the landing zone. Use the below resource links for more reading and implementation steps:

Copy data from/to a file system - https://learn.microsoft.com/en-us/azure/data-factory/connector-file-system
Create and configure a self-hosted integration runtime - https://learn.microsoft.com/en-us/azure/data-factory/create-self-hosted-integration-runtime
Copy new and changed files by LastModifiedDate - https://learn.microsoft.com/en-us/azure/data-factory/solution-template-copy-new-files-last-modified-date
Get Metadata activity in Azure Data Factory - https://learn.microsoft.com/en-us/azure/data-factory/control-flow-get-metadata-activity

I hope this is helpful. Please! Do not hesitate to let me know if you have any other questions, steps or clarifications.

Please do not close the thread by upvoting and accepting the answer if any part of it is helpful.

Answer 2

Srikanth Chavithina 265 Microsoft External Staff Moderator

Hi @Dwivedi, Upendra

Thanks for sharing the details.

What you're seeing is expected behavior. When a file is copied to a new location, the Created Date is updated, but the Modified Date usually stays the same. Since Azure Data Factory uses the Last Modified date for incremental file copy, files copied with an older Modified Date can be skipped.

Currently, the File System connector supports filtering only by Last Modified. It doesn't support filtering by Created Date.

Since using Get Metadata on a large number of files and nested folders can affect performance, one option is to keep a record of the files that have already been processed (for example, by storing their file name and path). Instead of checking only the Modified Date, you can compare new files against this record to identify files that haven't been processed yet.

Could you also let us know:

Is your network share an on-premises file share accessed through a Self-hosted Integration Runtime (SHIR)?
Approximately how many files and folders are being scanned in each pipeline run?

This information will help suggest the most suitable approach for your environment.

Dwivedi, Upendra 0 Reputation points

2026-07-18T15:59:00.3566667+00:00
Hi @Srikanth Chavithina ,

We are using SHIR to connect to on premised file share.

100+ files and all the deeply nested.

Users are updating old files as well by copy pasting data so making list of already ingested files and comparing new files is not sufficient.

Also this deeply nested folder structure will make this process a time taking process and problem is to reduce this processing time.
Srikanth Chavithina 265 Reputation points Microsoft External Staff Moderator

2026-07-18T16:43:03.2566667+00:00
Hi @Dwivedi, Upendra

Thank you for the additional details.

Based on what you've shared, I agree that keeping a list of previously processed files won't be enough, since users are also updating existing files by copying new data into them.

The challenge is that ADF's incremental file copy relies on the Last Modified timestamp. In your scenario, that timestamp isn't reliable for detecting both newly copied files and updates to existing files. Also, scanning a large number of files with Get Metadata can become slow when the folder structure is deeply nested.

Unfortunately, there isn't a native ADF feature that can efficiently detect both new and updated files for this type of file share based only on file timestamps.

Could you please clarify a couple of things?

Is the copy process done manually by users, or by an application or scheduled job?

Is it possible to change the source process so that new or updated files are copied into a separate landing folder or a list of changed files is generated before ADF runs?

Knowing this will help determine whether there's a more efficient approach than scanning the entire folder structure during every pipeline run
Dwivedi, Upendra 0 Reputation points

2026-07-18T17:22:24.0966667+00:00
@Srikanth Chavithina ,

Copy process was done by an application.

Unfortunately it is not possible.
Srikanth Chavithina 265 Reputation points Microsoft External Staff Moderator

2026-07-19T08:29:58.7966667+00:00

Hi @Dwivedi, Upendra
Thank you for the clarification.

Since the files are copied by an application and the copy process cannot be changed, this is an expected limitation of Azure Data Factory (ADF).

ADF's native incremental copy uses the Last Modified timestamp to detect new or changed files. If the application keeps the original Last Modified timestamp when copying the files, ADF cannot identify those files as new or updated.

In this case, the available options are to use a custom tracking method (such as a file manifest or metadata table) or another process outside of ADF to identify changed files before running the copy activity.

Unfortunately, there isn't a built-in ADF feature that can reliably detect files whose content has changed while the Last Modified timestamp remains unchanged.

I hope this helps clarify the current behavior. Please let us know if you have any further questions.

Thanks.
Srikanth Chavithina 265 Reputation points Microsoft External Staff Moderator

2026-07-20T08:35:27.2833333+00:00

Hi @Dwivedi, Upendra
I hope you had a chance to review the information shared earlier and found it helpful.

If you still have any questions or need further clarification, please let us know in the comments so we can assist you further. We look forward to hearing from you.
Dwivedi, Upendra 0 Reputation points

2026-07-20T09:07:37.0966667+00:00

Thankyou @Srikanth Chavithina for your help and i reviewed the information shared and i found it helpful.
Srikanth Chavithina 265 Reputation points Microsoft External Staff Moderator

2026-07-24T12:20:08.18+00:00

Hi @Dwivedi, Upendra
I'm following up regarding the response provided on your thread. Could you please review it and let us know if it addresses your question?

If the information resolves your issue, we would appreciate it if you could accept the response so it can help other community members with similar questions.