Struggling with Event Hub Errors: Unable to Trigger Functions After Changing Partition Count

Question

I have built a system where I send messages from devices to IoT Hub, pass them through Stream Analytics, and trigger Functions from Event Hub. I conducted a series of tests at this point and confirmed that Functions were triggered successfully.

Later, I decided to change the number of partitions in the Event Hub. To do this, I deleted one of the aliases of the Event Hub and recreated it with the same name but with a different partition count. However, after conducting tests following this change, I encountered the following error, and now I cannot trigger Functions from the Event Hub.

2023-07-24T09:31:54Z [Error] An exception occurred while receiving events for Event Hub: sensingedge-count (Consumer Group: '$Default', Partition Id: '0'); Operation Id: '8c6a5407-51b0-4348-9fc1-c6d32149aecb'. Error Message: 'The supplied offset '55846066024' is invalid. The last offset in the system is '-1' TrackingId:a3b6d45c-62a7-4f66-aa49-07c1cf7ed1f5_B5, SystemTracker:rl-eventhub-stg:eventhub:sensingedge-count~32766, Timestamp:2023-07-24T09:31:53 Reference:88685577-389e-46e1-8043-7ecfb9934651, TrackingId:478dfd8f-d20b-49cd-9a77-e233428681ce_B5, SystemTracker:rl-eventhub-stg:eventhub:sensingedge-count~32766|$default, Timestamp:2023-07-24T09:31:53 TrackingId:d1709763cb25457a8eefd0dd4cbc2a80_G7, SystemTracker:gateway5, Timestamp:2023-07-24T09:31:53 For troubleshooting information, see https://aka.ms/azsdk/net/eventhubs/exceptions/troubleshoot' 2023-07-24T09:31:54Z [Information] Completed receiving events for Event Hub: sensingedge-count (Consumer Group: '$Default', Partition Id: '0'); Operation Id: '8c6a5407-51b0-4348-9fc1-c6d32149aecb'. Service Retry Count: 1; Event Count: 0; Duration: '0.40' seconds 2023-07-24T09:31:54Z [Error] An exception occurred while processing events for partition '0' by processor instance with identifier '60231fe4-8ba6-4bb6-a172-0ab8be5ed8d5' for Event Hub: sensingedge-count and Consumer Group: $Default. Error Message: 'The supplied offset '55846066024' is invalid. The last offset in the system is '-1' TrackingId:a3b6d45c-62a7-4f66-aa49-07c1cf7ed1f5_B5, SystemTracker:rl-eventhub-stg:eventhub:sensingedge-count~32766, Timestamp:2023-07-24T09:31:53 Reference:88685577-389e-46e1-8043-7ecfb9934651, TrackingId:478dfd8f-d20b-49cd-9a77-e233428681ce_B5, SystemTracker:rl-eventhub-stg:eventhub:sensingedge-count~32766|$default, Timestamp:2023-07-24T09:31:53 TrackingId:d1709763cb25457a8eefd0dd4cbc2a80_G7, SystemTracker:gateway5, Timestamp:2023-07-24T09:31:53 For troubleshooting information, see https://aka.ms/azsdk/net/eventhubs/exceptions/troubleshoot'

Afterwards, I reverted to the original state, but the same error persisted, and the situation did not improve, and troubleshooting did not work for me. Could you please provide some guidance on how to address this issue?

Answer 1

Hello @KURAMOTOAHUJA TAIRA (倉本Ａ 平),

I see you are using the $default consumer group.

Consumer groups are not to be reused by multiple consumers (azure resources listening to incoming iot hub messages).

This could be Stream Analytics, Azure Functions, TimeSeries Insights, or even diagnostics tooling like Visual Studio Code or the IoT Explorer.

Because all these tools use $default by default, a mistake is made quickly.

Please use separate consumer groups for every consumer:

Check the Built-in endpoints page in the IoT Hub.

---

If the response helped, do "Accept Answer". If it doesn't work, please let us know the progress. All community members with similar issues will benefit by doing so. Your contribution is highly appreciated.

Answer 2

This question is pretty old, but it still pops up when searching for this issue on google. The accepted answer technically works, but not for the reason you might think and I think some additional explenation is helpful.

The Azure Functions Event Hubs trigger saves partition ownership and offsets in the storage account associated with the function. Look for a blob container called 'azure-webjobs-eventhub'. There should be a folder structure in it with event hub namespace name / event hub name / consumer group name, and in that folder there are blobs that track partition ownership and offsets. What went wrong here is that the event hub was recreated, and so the offset at the event hub side was reset. But the offsets in the blob storage still have the old offset, and because the name of the consumer group and event hub did not change it assumes that it can still use that. So it tries to consume starting at an offset that no longer exists.

After you change to a new consumer group, the function will create a new folder with partition ownership and offsets, and no longer use the old ones. That is why the accepted answer works. But is has nothing to do with the fact that using the default consumer group is a bad practice (even though it is). It is just because you switching consumer groups fixes the problem.

If you want to fix the problem without changing the consumer group, you can simply remove the folder for the consumer group in the storage container and then restart your function. Your function will simply recreate the folder structure and start consuming as if it was the first time it was deployed. Note that this might mean that the function will reprocess all messages on the event hub, depending on your settings.