We have a storage event trigger in adf kicking off pipeline to process file too early?

Question

We have a Storage event trigger, that is set to kick off a Data Factory Pipeline(To process files from blob container folder path ) onto a sql server database when a file is dropped on to the location.

The DataFactory Pipeline is a simple pipeline with Copy Activity to copy the parqet files to sql server tables.
The Storage event trigger is kicking off the pipleine before the file is finished being dropped so the copy activity runs and the data doesn't get transferred.

Other than adding a wait activity on the pipeline before kicking off the Copy Activity what is the best way to achieve this.

Answer 1

Hi Karkal,Prajna - Thanks for reaching out.

I would suggest you to review the operation that are happening on the account.
https://learn.microsoft.com/en-us/azure/storage/blobs/monitor-blob-storage?tabs=azure-portal

In case these are ADLS gen2 API (Create, Append and Flush) it is quite possible that the Event filtering is on both Create and Flush which could be leading to this situation.

The event might be triggering on Create while the file is still being written and it is then suggested to put the filtering on Flush File to ensure event is triggered only after file gets committed.

https://learn.microsoft.com/en-us/azure/event-grid/event-schema-blob-storage?toc=%2Fazure%2Fstorage%2Fblobs%2Ftoc.json&tabs=event-grid-event-schema

Hope that helps!

Please let me know if there are any further queries/concerns, will be glad to assist.

---

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.