adf pipeliine flow

Shambhu Rai 1,181 Reputation points

Hi Expert,

we are receiving files for every one mins in folder level .. how we can get the files loaded as soon as files receives

i.e. files receives every one mins... it should pick up and load in adls as soon as file receives


Azure Databricks
Azure Databricks
An Apache Spark-based analytics platform optimized for Azure.
1,716 questions
Azure Data Factory
Azure Data Factory
An Azure service for ingesting, preparing, and transforming data at scale.
8,535 questions
{count} votes

1 answer

Sort by: Most helpful
  1. Dillon Silzer 49,831 Reputation points

    Hello Shambhu,

    Have you set up a trigger for this? What type of trigger have you set up? I'd recommend setting up a trigger in response to storage events which works with ADLS gen2:

    Create a trigger that runs a pipeline in response to a storage event

    "The Storage Event Trigger currently supports only Azure Data Lake Storage Gen2 and General-purpose version 2 storage accounts. If you are working with SFTP Storage Events you need to specify the SFTP Data API under the filtering section too. Due to an Azure Event Grid limitation, Azure Data Factory only supports a maximum of 500 storage event triggers per storage account. If you hit the limit, please contact support for recommendations and increasing the limit upon evaluation by Event Grid team."

    Remember, it does take some time to run the pipeline after the trigger has began. Have you checked your Monitor logs to see when the pipeline fired and how long it took? I'd recommend checking your ADF Monitor logs for more information.

    If this is helpful please accept answer.