Using Azure Function with Event Grid Trigger to monitor and chunk Blob storage uploads.

Question

Hi,

I am interested in augmenting uploads to a Blob storage container in the following way:

1. Detect the file type of the file. If it is a docx or xlsx file, then convert them to pdf before upload.
2. When a pdf file is uploaded (included the ones that were freshly converted), detect the size of the file. If it is greater than or equal 16MB chunk, then chunk the pdf files into chunks of size 16MB or less.
3. Upload each chunk into the Blob storage container (or to an alternative Blob storage container.)

The ultimate goal is to have these chunks indexed to Cognitive Search and used in a Retrieval Augmented Generation pattern.

I was interested in using Azure Functions to detect changes to the Blob storage and perform the processing as necessary. Because my database is quite large, I am plan on using Event Grid triggers as I've read that it offers high-scalability with lower latency than blob trigger.

However, I can't seem to find resources or example code on how to use Event Grid to detect changes in my Blob storage. I am still fairly new to Azure Functions (only started within the last two days). May I have guidance on this matter? Please provide what output is expected if possible.

I am using VS Code for my testing and deployment of Azure Function app.

Thanks!

Answer 1

Hello, @johananmahendran - Welcome to Azure, and thanks for reaching out!

You're on the right path by going with the Event Grid trigger, and here's how I would break down the workflow:

• Configure Event Topic and Event Subscription by following the steps in this doc: Reacting to Blob storage events
  • Be sure to thoroughly review all details like Event Schema and supported Event types.
  • Use Event filters for .docx or .xlsx files.
• Setup Azure Functions with Event Grid trigger:
  • How to work with Event Grid triggers and bindings in Azure Functions
  • Azure Event Grid bindings for Azure Functions
• Use Azure Blob SDK (available for languages other than c#) for handling chunks, creating new .pdf blobs & relevant libraries for the language of your choice to handle conversion.

Additionally, I would also consider Logic Apps for this type of workflow as it might provide quicker time-to-solution using a declarative, designer-first development approach.