Why does my queue triggered Azure function occasionally stop working?

Hi Steve, First, my sincere apologies for the delayed response thank you for your patience and for providing such detailed updates while monitoring the situation. I really appreciate you keeping such thorough notes; they’re incredibly helpful for troubleshooting. Let me address each of your messages:

1. Downtime events (Sep 18–19)
   • lbrth-prod-aue-fnc: Stopped processing between 7:44 pm–11:07 pm NZT on Sep 18, resuming after your restart pipeline at 11 pm.
   • lbrth-prod-aue-shared-scheduled-fnc: Brief pause between 5:02 am–5:56 am NZT on Sep 19, recovered without manual restart. These point to the runtime briefly not polling the queue exactly the intermittent pauses we’ve been trying to eliminate.
2. Degraded performance after the new host.json
   • I understand that after applying the recommended V2-style host.json to lbrth-prod-aue-fnc, slides began taking >1 hr to generate and queue items backed up—enough to affect customers so you wisely reverted at 11:35 am NZT on Sep 19. The other two functions remain on the updated host.json without visible issues.
3. Why the updated host.json could cause backlog Based on what you shared and what I can see from backend defaults:
   • NewBatchThreshold setting: The value of 16 we originally suggested may have been too low for your workload. If the runtime waits until fewer than 16 in-flight messages remain before fetching a new batch, it can underutilize available throughput, especially under heavy load. Your reverted file’s newBatchThreshold: 250 is much more aggressive.
   • MaxPollingInterval: Your reverted file uses 00:00:02 (poll every 2 sec), while the recommended default maxPollingInterval was not explicitly set so the runtime may have polled far less frequently, allowing the queue to build up.
   • VisibilityTimeout difference: Reverted value 00:00:30 keeps items hidden only briefly helpful for quick retries. The previous 00:03:00 may have held items too long when processing stalled.
   • Message Encoding case sensitivity: The runtime is case-insensitive for "base64", but mismatches sometimes show odd behavior across bindings. So, the combination of slower polling + lower batch threshold almost certainly throttled throughput.

Next steps / Suggestions

1. Stay on the reverted host.json for lbrth-prod-aue-fnc for now since it’s performing well.
2. For the other two functions on the new file, consider aligning maxPollingInterval and newBatchThreshold closer to the reverted values to match your load pattern.
3. When you have a non-production window, you could try a hybrid host.json like:

      {
        "version": "2.0",
        "functionTimeout": "00:10:00",
        "extensions": {
          "queues": {
            "maxPollingInterval": "00:00:02",
            "visibilityTimeout": "00:00:30",
            "batchSize": 32,
            "maxDequeueCount": 5,
            "newBatchThreshold": 250,
            "messageEncoding": "base64"
          }
        }
      }
      
   

   This keeps the V2 structure (to avoid parsing issues) but preserves your proven performance settings.
4. After testing, monitor queue length vs. function executions in the Azure portal (Storage Queue Metrics and Function Invocation Count). If the hybrid config stabilizes behavior, we can standardize it across your apps.

References:

• host.json queues configuration (v2+)
• Queue trigger scaling behavior

Again, apologies for the late follow-up, and thank you for all the precise timestamps and rollback details they make diagnosing much easier. Please let me know if you’d like me to pull backend telemetry for those exact downtime windows. Please let me know if you have further questions. Thank you!