Azure Functions with Event Hub Trigger - Not able to caught up on incoming messages load

Question

@Ryan Hill I was following an earlier post(https://learn.microsoft.com/en-us/answers/questions/1331841/azure-functions-with-event-hub-trigger-incoming-me) related to my question

I am also facing similar kind of issue wherein my event hub trigger is not able to caught up on the incoming messages load.

For my use case -

We have Event Hub which is on dedicated tier and has 64 partition count and one consumer group. My function app is on isolated app service plan(Isolated v2 I2V2) and running 8 instances.

I am using below settings in host.json for my event hub trigger

"eventHubs": {

  "maxEventBatchSize": 2048,

  "batchCheckpointFrequency": 5,

  "prefetchCount": 4096

},

I do see all my 8 instances are working but still it's not able to caught up on the incoming messages load. We are getting around 5 million events per day and my function app is only able to process <1 million.

I have below questions.

Q1 - How do I know how many concurrent executions are happening per instance and is there any way to increase concurrent executions?

Answer 1

Thank you for reaching out @Malik, Deepankar.

Event Hub trigger uses the Event Processor Host under the covers which takes care of the concurrency by creating instances according to the number of partitions, however, depending on the throughput that you want to achieve and the processing being done for each batch of events, 8 instances owning 8 partitions each might not be enough.

There are a few things to check:

1. Are the function app instances having any sort of backpressure?
2. Are there downstream dependencies as part of the event processing having backpressure?
3. Can you test by scaling out the number of instances to 16?

Regarding your question on how to monitor the executions, the trigger can integrate with Application Insights for you to view each execution, see this documentation for more details, you can then query and perform aggregates in order to understand how many concurrent executions are taking place as well as how many events are being pulled per execution.

Check sample Application Insights queries for more details/examples on how to query this data

Answer 2

Hello Malik, Deepankar

You can check the number of concurrent executions happening per instance by going to the "Monitor" tab in the Azure portal and selecting your Function app.

From there, you can view the "Function Execution Count" chart, which will show you the number of executions happening over time. Regarding increasing concurrent executions, you can scale out your Function app by increasing the number of instances running. Since you are already running 8 instances, you may want to consider optimizing your function code to improve its performance.

You can also try increasing the number of partitions in your Event Hub to distribute the load across more partitions, which can help improve the overall throughput.

Additionally, you may want to review the settings in your host.json file to ensure they are optimized for your use case. The settings you provided seem reasonable, but you may want to experiment with different values to see if they improve performance.

I hope this helps