About Disaster Recovery on Event Hubs + Functions

Jona 455 Reputation points
2024-06-06T17:32:30.5533333+00:00

Hi,

I'm facing an architectural challenge. I have:

  • A Blob Storage where files will be landing
  • A Function environment where one functions gets executed every time a file is uploaded. This is done via Event Grid notification source on Blob trigger
  • An Event Hubs namespace where a destination event hub resides.

I'm already familiar to disaster recovery options, geo redundancy/replication. However, I need to implement some logics in a PoC to achieve some requirements from architect team.

The requirement:

  • We have a stream of files with an unpredictible workloads and peaks. Sometimes, files arrives with a size of some KB's a others with dozens of MB's. The max file size that has arrived is 150MB
  • We have only 5 TU in our Standard Event Hub namespace
  • During the PoC, we will "shut down" Event Hubs

So, I've implemented (or I have in mind):

  1. A Function that takes the landed file anc checks the file size. If needed, the file is splitted on smaller pieces to fit the 1MB publication limit on Event Hub. Those pieces are store on a different container for a another Function to sent it to EVent Hub
  2. The Function sending the pieces to Event Hub must check that Event Hub is up and running. I know that this can be mitigated with DR and Redundancy features. However, I've been asked to check the availabilty of Event Hub ... ¿how can this be achieved?
  3. If Event Hub is not up, in order no not queu many Functions, I want to implement that the files/pieces to be uploaded to another container. That way I terminate the Function execution. If a Function has an output binding (event hub) and the output service is not up... ¿how can I terminate the Function?, ¿What the return value should be?

Any opinion would be appreciated

Regards

Jona

Azure Functions
Azure Functions
An Azure service that provides an event-driven serverless compute platform.
4,476 questions
Azure Event Hubs
Azure Event Hubs
An Azure real-time data ingestion service.
579 questions
0 comments No comments
{count} votes