best practice

Question

best practice

arkiboys 8,066

Hi,
What is the best practice to execute the appropriate pipeline(s) with parameters based on the file(s) being landed in container A?
Thank you

Answer 1

Hello @arkiboys ,

Thanks for the question and using MS Q&A platform.

My understanding is that you would want to auto trigger a pipeline(s) run when a file(s) landed in an azure storage container. If that is the case, then you can utilize the in-built functionality called Storage Event Triggers.

You can configure to trigger pipelines based on events happening in storage account, such as the arrival or deletion of a file in Azure Blob Storage account. Data Factory and Synapse pipelines natively integrate with Azure Event Grid, which lets you trigger pipelines on such events.

Here is a demonstration video which show how to configure and use these Storage Event Triggers: Event based Triggers in Azure Data Factory

Please note below limitations with Storage Event triggers:

The Storage Event Trigger currently supports only Azure Data Lake Storage Gen2 and General-purpose version 2 storage accounts. Due to an Azure Event Grid limitation, Azure Data Factory only supports a maximum of 500 storage event triggers per storage account. If you hit the limit, please contact support for recommendations and increasing the limit upon evaluation by Event Grid team.

Hope this will help. Please let us know if any further queries or if I misunderstood your requirement. And if you have further query, please elaborate more about the exact use case scenario so that we can better assist you.

------------------------------

Please don't forget to click on or upvote button whenever the information provided helps you. Original posters help the community find answers faster by identifying the correct answer. Here is how
Want a reminder to come back and check responses? Here is how to subscribe to a notification
If you are interested in joining the VM program and help shape the future of Q&A: Here is how you can be part of Q&A Volunteer Moderators

arkiboys 8,066 Reputation points

2022-01-21T07:32:23.32+00:00

Thank you for the reply, and I agree to go through the root of event driven triggers, however my requirement has more to it as follows:
Once the file(s) has arrived, somehow I would like to know which pipeline to be executed for that very file(s) and which config file to use to read connection, etc.

My current thoughts to do one of the following methods:
1-
to have a config file in the blob storage. each time a file is landed and the event trigger is fired then the standard pipeline simply reads the config file to see what it has to do with the arrived file. And then it can execute other pipeline as per config file...

2-
First have a blank file which has nothing in it. It lands in the input blob storage container. This blank file has a name which the name shows, the name of the pipeline to be executed and the config file name to be read and perhaps some other information. All these informations are in the name of the file i.e. pipeline12,customerContacts.csv,config12.trg
The sample filename above has no information inside the file but it has information in the filename as follows:
pipeline12 --> indicates which pipeline to be executed
customerContact.csv --> indicates the .csv file has arrived
config12 --> indicates which config file to read to get the source location of the .csv file
The trg is the extension of the file which has these names in it

I feel option 1 is a better solution and option 2 is challenging as I have to split out the details separately, etc.

What do you think?

Thank you
KranthiPakala-MSFT 36,022 Reputation points Microsoft Employee

2022-01-21T20:14:08.977+00:00

Hello @arkiboys ,

Yes, I do agree with approach1. The implementation will be clean and straightforward when compared to approach 2.
Feel free to open a new thread if you face any issues while implementing the solution. We will be happy to assist.

Have a good day! :)

best practice

0 additional answers

Activity