Real-time Processing of High-volume Requests in Azure Functions: Reducing Latency in IoT Hub and Event Hub Integration

KURAMOTOAHUJATAIRA-3118 250 Reputation points
2023-07-28T06:32:25.6966667+00:00

Please advise on how to process a large number of requests in real-time using Azure Functions. I have built a system that sends messages from devices connected to IoT Hub and triggers Functions through Event Hub. I conducted a load test with the following configurations:

  • Functions Plan: Premium v3 P1V3
  • Load Test Content: Sending 400 messages per second from devices to IoT Hub for 30 seconds.
  • Event Hub Scaling Settings: Throughput Units × 4, Automatic inflation Max 40
  • Functions Scaling Settings: 10 instances (Manual)
  • Execution Time: 30 seconds between 01:31 and 01:32

Issue: As observed, Event Hub finishes outputting messages around 01:35, but Functions continue executing until around 01:42. In other words, even though all messages from the devices to IoT Hub are sent by 01:32, it takes about 3 minutes for Event Hub's output to complete and 10 minutes for Functions to finish processing.

Event Hub outgoing:
User's image

Function execution count:
User's image

If this were a longer-term test, I believe the delay until Functions' processing completion would increase further. Ideally, I would like the delay between the last message being sent to IoT Hub and the completion of Functions processing to always be within 1 minute.

While it is not strictly necessary to reduce the delay to 1 minute, I am concerned that the delay time increases with the number of messages sent. I initially thought scaling Functions might resolve the issue, but it doesn't seem to work as expected. I would appreciate any advice you can provide.

Azure Functions
Azure Functions
An Azure service that provides an event-driven serverless compute platform.
4,006 questions
Azure IoT Hub
Azure IoT Hub
An Azure service that enables bidirectional communication between internet of things (IoT) devices and applications.
1,083 questions
Azure Event Hubs
Azure Event Hubs
An Azure real-time data ingestion service.
530 questions
0 comments No comments
{count} votes

Accepted answer
  1. Sander van de Velde 26,811 Reputation points MVP
    2023-07-28T14:36:41.3+00:00

    Hello @KURAMOTOAHUJA TAIRA (倉本A 平)

    you rely on several techniques for which the impact is hard to measure (like automatic inflation).

    Normally, it takes several seconds or even minutes before the underlying architecture is updated toward the new scaling abilities.

    Give Azre a chance to 'warm up'.

    I recommend executing this test multiple times and taking averages. You probably get more meaningful results if you set everything at a fixed scale.

    The size of the messages has an impact due to the number of units (thus MB/s).

    The number of partitions is mainly related to the number of devices. 4 partitions are good enough for most customers.

    Just for comparison, you could also leave out the event hub and use the Azure Functions with the Azure IoT Hub trigger.


    If the response helped, do "Accept Answer". If it doesn't work, please let us know the progress. All community members with similar issues will benefit by doing so. Your contribution is highly appreciated.

    0 comments No comments

1 additional answer

Sort by: Most helpful
  1. Pramod Valavala 20,501 Reputation points Microsoft Employee
    2023-07-28T11:06:25.9933333+00:00

    @KURAMOTOAHUJATAIRA-3118 Here are a few things that I believe could help here

    1. If your function is receiving events in batches, you could increase the batch size to process more messages in each invocation in host.json
    2. When consuming messages from Event Hubs, only one instance of your function app can acquire a lease on an Event Hub partition. Since you are manually scaling your function app to 10 instances, ensure you have 10 partitions on your Event Hub as well.
    3. Like the previous, you could increase the number of partitions and your function app instances to handle further load.
    4. Ensure you are using async code where possible
    1 person found this answer helpful.