Azure service buss triggered Functions always miss a few execution

Question

I have a service bus queue triggered azure function (python), which processes around 5000 messages per night.

Every night there are always around 5-10 messages enter the dead-letter queue.

I checked in the function application insight panel and I could not find the executions for these 5-10 messages.

It seems like these messages do not trigger a function, although the queue has a max-delivery count of 5, and then go directly into dead-letter. Any idea why?

The host.json of my function.

{
  "version": "2.0",
  "logging": {
    "applicationInsights": {
      "samplingSettings": {
        "isEnabled": true,
        "excludedTypes": "Request"
      }
    }
  },
  "extensionBundle": {
    "id": "Microsoft.Azure.Functions.ExtensionBundle",
    "version": "[3.*, 4.0.0)"
  },
  "concurrency": {
    "dynamicConcurrencyEnabled": false,
    "snapshotPersistenceEnabled": false
  },
  "extensions": {
    "serviceBus": {
      "prefetchCount": 10,
      "maxConcurrentCalls": 5,
      "autoCompleteMessages": true,
      "maxAutoLockRenewalDuration": "00:10:00"
    }
  },
  "functionTimeout": "01:00:00"
}

Answer 1

@Bin Jin (Forestay) Thanks for reaching out. If there are messages that goes into the dead letter queue then to find the reason for dead letter queue please peek those messages from dead letter queue using service bus explorer to know the reason why the messages were moved to the dead letter queue as there can be multiple reason as mentioned here.

Looking into the configured most probably the reason for those message moved to the dead letter queue would be MaxDeliveryCountExceeded reason as the messages was not consumed by your client application i.e. Azure function.

If I look into the configuration, then I can see you have define the prefetchCount as 10 so that means 10 messages will be prefetched at the client buffer in one call so there could be chances that the client was not able to process those messages within the lock duration as the client application would be in high utilization. I will suggest you to first set the maximum lock duration as 5 minutes in your entity (i.e. queue) and set the maxAutoLockRenewalDuration to 4 minutes and decrease the prefetchCount to some other value or remove it as per your load.

To know more details on the prefetch please refer to this document and to know more about service bus lock please refer to this documenation.

Feel free to get back to me if you have any queries or concerns.