Azure Functions with Event Hub Trigger - Incoming Messages Is Much Higher than Outgoing Messages

Question

Azure Functions with Event Hub Trigger - Incoming Messages Is Much Higher than Outgoing Messages

Amir Zuker (CV) 75

Hi,

We have an Azure Function App with a set of functions and most of them are connected to different event hubs via Event Hub trigger.

The problem:

There's a specific function that runs at lower rate than the actual workload reaching the event hub.
Note that we have auto scale enabled with maximum instances on the consumption tier, plus the specific event hub has only one consumer group and the only consuming party is that specific azure function.

Findings that support it:

The incoming messages in the event hub is bigger than the outgoing messages as you can see in the following image. Note that we have no idea what happened on the date where these metrics started diverting, it's unclear why that happened.

User's image

Further investigations:

We assume we hit some kind of quota imitation and there's some sort of throttling perhaps.

We did see that there was throttling on the event hub namespace by observing the "Throttled Requests" metric.
We increased the throughput units and enabled the auto inflate feature as well which brought "Throttled Requests" metric to zero, which is good, but the gap between the incoming and outgoing remains the same.

Perhaps we're hitting function scale quota? Perhaps even at subscription level limits as mentioned here - https://learn.microsoft.com/en-us/azure/azure-functions/functions-scale#scale

Problem is, it is unclear how we can diagnose and troubleshoot it.

Additional context:

Function app is in a consumption tier
Function app has auto scale enabled, set to maximum (200 instances)
Function app includes other functions as well, all other functions seem to operate as expected, only this function in specific seems to have a problem (all other functions are connected to different event hubs but on the same event hub namespace)
Event hub namespace is in a standard tier
As mentioned, this specific azure function is the only consuming party of this specific event hub, which has only one consumer group
Azure function uses cardinality set to many

Can anyone please help?
When can the incoming messages in event hubs be larger than outgoing when "Throttled Requests" metric is zero and the single consuming party is set up with auto-scale?
How can we diagnose and troubleshoot it further?

Amir Zuker (CV) 75 Reputation points

2023-07-19T13:22:42.96+00:00

We made some progress.

The limit we were facing seems to be the maximum horizontal scale we could achieve in that specific setting. When we increased the event hub partition count from 2 to 4, the outgoing throughput was increased and the gap has reduced.

It seems we need to do some fine-tuning over the partition number and perhaps configuration related to batching/prefetch/checkpointing too.
Relevant read to whoever is interested - https://learn.microsoft.com/en-us/azure/architecture/serverless/event-hubs-functions/performance-scale

Problem is, changing the partition number isn't a simple change as it requires recreating the event hub when it is in standard tier. We have IoT hub inserting to this event hub, and changing that causes issues where we need to rebind things (both in IoT hub and azure function in our case). Additionally, it may lead to duplicates too if we need to reset checkpoints.

We can upgrade the tier towards premium or dedicated cluster, however that is too costly for us at this point.
We are currently evaluating the option of using Azure Service Bus instead.

Note - there was a lot of trial and error here. It would have been nice if specific this limit situation could be more visible and better diagnosed.
Ryan Hill 30,281 Reputation points Microsoft Employee Moderator

2023-07-20T19:40:25.1166667+00:00

Wait @Amir Zuker (CV) just to confirm, only one of functions within your function app is experiencing this slow down. When it comes to End-to-end transaction details, there any patterns that you're noticing; increase time spent in the queue, message timeouts, increased execution time of the function?

It may be best to work offline. Please email at AzCommunity[at]microsoft.com ATTN: Ryan with a link to this post and the names of your function app (and the function itself) and event hub.

Your answer

Amir Zuker (CV) 75 Reputation points

2023-07-19T13:22:42.96+00:00

We made some progress.

The limit we were facing seems to be the maximum horizontal scale we could achieve in that specific setting. When we increased the event hub partition count from 2 to 4, the outgoing throughput was increased and the gap has reduced.

It seems we need to do some fine-tuning over the partition number and perhaps configuration related to batching/prefetch/checkpointing too.
Relevant read to whoever is interested - https://learn.microsoft.com/en-us/azure/architecture/serverless/event-hubs-functions/performance-scale

Problem is, changing the partition number isn't a simple change as it requires recreating the event hub when it is in standard tier. We have IoT hub inserting to this event hub, and changing that causes issues where we need to rebind things (both in IoT hub and azure function in our case). Additionally, it may lead to duplicates too if we need to reset checkpoints.

We can upgrade the tier towards premium or dedicated cluster, however that is too costly for us at this point.
We are currently evaluating the option of using Azure Service Bus instead.

Note - there was a lot of trial and error here. It would have been nice if specific this limit situation could be more visible and better diagnosed.
Ryan Hill 30,281 Reputation points Microsoft Employee Moderator

2023-07-20T19:40:25.1166667+00:00

Wait @Amir Zuker (CV) just to confirm, only one of functions within your function app is experiencing this slow down. When it comes to End-to-end transaction details, there any patterns that you're noticing; increase time spent in the queue, message timeouts, increased execution time of the function?

It may be best to work offline. Please email at AzCommunity[at]microsoft.com ATTN: Ryan with a link to this post and the names of your function app (and the function itself) and event hub.

Share via

Azure Functions with Event Hub Trigger - Incoming Messages Is Much Higher than Outgoing Messages

Your answer