Azure Function EventHub Trigger stops consuming events

Question

I have an Azure Function EventHub trigger running in a Linux Consumption Plan which keeps stopping processing of events. There are no errors, events are still coming in, it's just like the Function goes to sleep. If I go into the Function App in the Portal, then it starts processing again - I don't change anything. But it will then stop again.

• West Europe
• Using a custom consumer group, not the $default consumer group
• Using a geo-paired eventhub, so using the alias connection string
• Storage account general purpose V2 RA-GZRS, does not have hierachical namespace enabled, and does not have Data Lake Storage enabled
• Availability and Performance -> "Functions that are not triggering" in the portal shows a "Host Runtime instance (Dynamic Plan) was not available for a long time period" warning
• Going into the portal for the function always kicks it off and events then get processed but obv not a feasible solution for production

Update:

• enabled Scale Controller logging as per https://learn.microsoft.com/en-us/azure/azure-functions/configure-monitoring?tabs=v2#configure-scale-controller-logs
• checked those logs and can see a Severity 2 log:
  "FunctionName: '<MyFunctionName>'. Trigger's Event Hub name '%EventHubSettings:HubName%' failed to resolve from AppSettings. Error: ''%EventHubSettings:HubName%' does not resolve to a value.'.
  • the trigger function method is:
    public async Task Run([EventHubTrigger("%EventHubSettings:HubName%", Connection = "EventHubSettings:Connection", ConsumerGroup = "%EventHubSettings:ConsumerGroup:MyCustomConsumerGroup%")]
  • those app settings have been configured in the portal

Answer 1

I think I've got to the bottom of this.

Cause:
The behaviour exhibits itself if you use appsetting substitution syntax like below, where your appsettings are nested
[FunctionName("TestEventHubTrigger")]
public async Task Run([EventHubTrigger("%EventHubSettings:HubName%", Connection = "EventHubSettings:Connection", ConsumerGroup = "%EventHubSettings:ConsumerGroup%")]

The ScaleController logs out about being unable to resolve the values (per my Update in the question), and so does not end up monitoring the EventHub for messages to know when to scale out the instances.

Resolution:
Change the appsettings to be at the root level, not nested and change the function signature to:
[FunctionName("TestEventHubTrigger")]
public async Task Run([EventHubTrigger("%EventHubSettingsHubName%", Connection = "EventHubSettingsConnection", ConsumerGroup = "%EventHubSettingsConsumerGroup%")]

No more warnings in ScaleController log output, and the function no longer gets stuck in permanent sleep (low event-rate scenario). You also see an increase in the "Requests" metric on the EventHub instance, which seems to be a sign that the ScaleController is properly monitoring the trigger.

I've blogged a fuller write-up: https://www.adathedev.co.uk/2021/12/troubleshooting-azure-function.html

Answer 2

Functions Apps with Consumption Plan (Serverless) will sleep when they haven’t been called recently (5 minutes of inactivity). And they will have a cold start, meaning when they are waking up to serve, it's going to take some time. But this doesn't mean, you need to navigate to the Function App in the Azure Portal to wake it up always. Whatever the function's triggers should wake it up.

Can you try moving the Function App to a dedicated plan and see whether it's working as expected?
https://learn.microsoft.com/en-us/azure/azure-functions/functions-scale#cold-start-behavior