Real-time Processing of High-volume Requests in Azure Functions: Reducing Latency in IoT Hub and Event Hub Integration

Question

Please advise on how to process a large number of requests in real-time using Azure Functions. I have built a system that sends messages from devices connected to IoT Hub and triggers Functions through Event Hub. I conducted a load test with the following configurations:

• Functions Plan: Premium v3 P1V3
• Load Test Content: Sending 400 messages per second from devices to IoT Hub for 30 seconds.
• Event Hub Scaling Settings: Throughput Units × 4, Automatic inflation Max 40
• Functions Scaling Settings: 10 instances (Manual)
• Execution Time: 30 seconds between 01:31 and 01:32

Issue: As observed, Event Hub finishes outputting messages around 01:35, but Functions continue executing until around 01:42. In other words, even though all messages from the devices to IoT Hub are sent by 01:32, it takes about 3 minutes for Event Hub's output to complete and 10 minutes for Functions to finish processing.

Event Hub outgoing:

Function execution count:

If this were a longer-term test, I believe the delay until Functions' processing completion would increase further. Ideally, I would like the delay between the last message being sent to IoT Hub and the completion of Functions processing to always be within 1 minute.

While it is not strictly necessary to reduce the delay to 1 minute, I am concerned that the delay time increases with the number of messages sent. I initially thought scaling Functions might resolve the issue, but it doesn't seem to work as expected. I would appreciate any advice you can provide.

Answer 1

Hello @KURAMOTOAHUJA TAIRA (倉本Ａ 平)

you rely on several techniques for which the impact is hard to measure (like automatic inflation).

Normally, it takes several seconds or even minutes before the underlying architecture is updated toward the new scaling abilities.

Give Azre a chance to 'warm up'.

I recommend executing this test multiple times and taking averages. You probably get more meaningful results if you set everything at a fixed scale.

The size of the messages has an impact due to the number of units (thus MB/s).

The number of partitions is mainly related to the number of devices. 4 partitions are good enough for most customers.

Just for comparison, you could also leave out the event hub and use the Azure Functions with the Azure IoT Hub trigger.

---

If the response helped, do "Accept Answer". If it doesn't work, please let us know the progress. All community members with similar issues will benefit by doing so. Your contribution is highly appreciated.

Answer 2

@KURAMOTOAHUJATAIRA-3118 Here are a few things that I believe could help here

1. If your function is receiving events in batches, you could increase the batch size to process more messages in each invocation in host.json
2. When consuming messages from Event Hubs, only one instance of your function app can acquire a lease on an Event Hub partition. Since you are manually scaling your function app to 10 instances, ensure you have 10 partitions on your Event Hub as well.
3. Like the previous, you could increase the number of partitions and your function app instances to handle further load.
4. Ensure you are using async code where possible