adf pipeliine flow

Question

Hi Expert,

we are receiving files for every one mins in folder level .. how we can get the files loaded as soon as files receives

i.e. files receives every one mins... it should pick up and load in adls as soon as file receives

Regards

Answer 1

Hello Shambhu,

Have you set up a trigger for this? What type of trigger have you set up? I'd recommend setting up a trigger in response to storage events which works with ADLS gen2:

Create a trigger that runs a pipeline in response to a storage event

https://learn.microsoft.com/en-us/azure/data-factory/how-to-create-event-trigger?tabs=data-factory

"The Storage Event Trigger currently supports only Azure Data Lake Storage Gen2 and General-purpose version 2 storage accounts. If you are working with SFTP Storage Events you need to specify the SFTP Data API under the filtering section too. Due to an Azure Event Grid limitation, Azure Data Factory only supports a maximum of 500 storage event triggers per storage account. If you hit the limit, please contact support for recommendations and increasing the limit upon evaluation by Event Grid team."

Remember, it does take some time to run the pipeline after the trigger has began. Have you checked your Monitor logs to see when the pipeline fired and how long it took? I'd recommend checking your ADF Monitor logs for more information.

---

If this is helpful please accept answer.