Azure function event hub trigger failure

Question

Recently I see some event hub trigger failure that stop function completely. Only way I can start processing again is by switching to another consumer group. Restart function app or disable/enable function does not resolve the problem. Function would try to acquire lease with event hub, then end up with failure. Blob checkpoint shows lease expired. Function assemblies become inaccessible until I disable the event hub trigger. I have collected some errors happened within 10 minutes before trigger completely stopped.

34 user errors are recorded on event hub.

In function insights, 2 types of errors are recorded.

2 of these errors:

The lease ID specified did not match the lease ID for the blob

Then, 10 of these errors:

Ignoring out of date checkpoint with offset 8585647668056/sequence number 39975337 because current persisted checkpoint has higher offset 8585647777424/sequence number 39975386 (Parameter 'offset/sequenceNumber')

Answer 1

I have this issue fixed. The reason is because I used Task.Result which caused deadlock. It’s explained in this blog.
https://blog.stephencleary.com/2012/07/dont-block-on-async-code.html
Based on my testing on function premium plan, the problem only happen when more partitions are mapped to single instance. 1 or 2 partitions per instance are not affected. That's why it happen when function scale in as less data are flowing.
After I recoded the logic, trigger works perfectly.
I think if Azure function can provide a alert on non returned function, it would be very helpful to troubleshooting similar issue.