Copy the files from Storage account using ADF before the source files are completely copied over ?

Sid_1805 91

Use case : The ADF pulls up a 0KB file from Storage Account because the file was in progress of being copied and not completed in the source.
It is picking up the file as valid even before it is completely copied over in the Storage Account.

How to put a check in ADF to ensure the file has completed its transaction of being copied over and not copied over in between.
Thanks

Subashri Vasudevan 11,206 Reputation points

2022-10-16T12:25:44.48+00:00

Hi @SiddharthaKrishna-8267

Do you have storage trigger or how is the pipeline identifying the file?
Subashri Vasudevan 11,206 Reputation points

2022-10-18T15:51:09.457+00:00

Hi @Sid_1805

Just checking to see if you need some help here.

Thanks
Sid_1805 91 Reputation points

2022-10-19T06:05:28.123+00:00

@Subashri Vasudevan - ADF currently copied the files from Azure Files ( X location ) to Azure Storage.

What is happening that before the copy operation to the azure storage is completed , the downstream 3rd party system can read the file metadata of an in-complete file and it triggers the downstream loads.

Looking for a way to lock the copy operation transaction so that there are no Ready Reads downstream.

Thank you
Subashri Vasudevan 11,206 Reputation points

2022-10-19T09:51:50.727+00:00

Hi @Sid_1805 ,

There's no straight forward process to lock files.

Please check similar thread

As suggested in the other thread, you can download the file or copy the file to a different folder (which is not looked upon by the downstream)

And once the copy activity completes, you can have one more copy activity to copy the file from temp folder to actual folder looked on by the downstream system.

Let us know it it helps.
Sid_1805 91 Reputation points

2022-10-22T20:27:12.457+00:00

@Subashri Vasudevan - That is quite strange to know we cannot prevent a dirty read in File copy transfers using ADF :(
Having different folders wont help as the downstream Job would poll any given path on real time basis.

Exclusive lock operation during write should be a must have feature !!

thanks for confirming :)
Subashri Vasudevan 11,206 Reputation points

2022-10-23T02:02:13.13+00:00

hello @Sid_1805

Here is a discussion on similar lines.

Did you go through before ?

Thanks

Share via

Copy the files from Storage account using ADF before the source files are completely copied over ?

Your answer