Send Email when file lands in Azure Data Lake Storage Gen 2

Question

Send Email when file lands in Azure Data Lake Storage Gen 2

pm-syn-lover 36

Currently using Synapse to process data, and I have a folder in my Azure Data Lake Storage Gen 2 that is called 'Failed'. When any bad data is processed and placed in 'Failed' I want to send an email notification. I was thinking I could do this with Logic Apps but I'm not seeing that capability. Any suggestions?

Thank you!!

2 answers

Your answer

Answer 1

Michael Taylor 60,326

You can do this with a durable function which is a specialized Azure Function. Logical apps can also do it. The differences boil down to how much integration and customization you want. Refer to this doc on the official differences.

In both cases you define a storage trigger that monitors the container you care about. When a blob is added or updated in the container then the trigger is fired and Azure spins up your DF or LA.

Create a new DF/LA in Azure. The recommendation is to use consummation model.
The DF/LA will either directly contain the logic to process the request or call out to an API that does.
Add a storage trigger that points to the container you want to monitor. For logical apps you can see the instructions here.

pm-syn-lover 36 Reputation points

2021-08-24T16:30:05.667+00:00

@Michael Taylor Thank you for the quick response! Do you know if you can customize to where it looks at only one specific directory?
Michael Taylor 60,326 Reputation points

2021-08-24T17:45:48.52+00:00

This is a common misconception for Azure Storage. There is no such thing as a folder/directory in containers. A container is just a list of blob data with names. A blob name can have slashes in it but that doesn't change anything. For example a container can contain both a blob named 'something\someone' and 'someone'. There are 2 separate blobs in the container and they are flat. Some UIs (such as the Azure Storage Preview) do assume that you're building a "folder-like" structure and therefore will show a directory structure but under the hood there is just the single blob file. That is why you cannot have empty "folders" in a container.

So, to answer your question, yes you can limit it to a "folder" by simply specifying the file mask when defining the trigger. An example is given here where you filter on blobs that have a name starting with original-. What I don't know is if it'll like a pattern that includes slashes in the name (outside the container). But you could try container/folderA/{name} which would in theory match files that begin with the name folderA/.
Michael Taylor 60,326 Reputation points

2021-08-24T17:46:51.133+00:00

If you cannot get the filtering to work at the trigger level then you can also just use the base trigger to get notified and then in your function activity check for a more complex name and abort if needed. It requires consumption but would still allow the use of functions.

Answer 2

Pramod Valavala 20,656 Microsoft Employee Moderator

@pm-syn-lover Azure Data Lake Storage Gen 2 supports events via Azure Event Grid that can be used to trigger a logic app that sends your email using the outlook connector.

With Azure Event Grid, you can also filter events to the ones that you are interested in.

Share via

Send Email when file lands in Azure Data Lake Storage Gen 2

2 answers

Your answer