Queue trigger function stopped without (visible) reason and long delay before next try

Question

Hello,

we have a queue triggered function on consumption plan, which sometimes stops while running.

We want to process 1 message / instance, here are the queue settings in host.json:

 "maxPollingInterval": "00:00:02",
 "visibilityTimeout": "00:00:05",
 "batchSize": 1,
 "maxDequeueCount": 2,
 "newBatchThreshold": 0

A message was written to the queue at 3:54:56 PM.

We found the following logs

at

at 9/9/2024, 3:58:30.6648752 PM [HostMonitor] Host CPU threshold exceeded (95 >= 80)

at 9/9/2024, 4:05:09.2535375 PM Trigger Details: MessageId: 0dfa5a35-f122-4db3-9300-6007ca64194e, DequeueCount: 2, InsertedOn: 2024-09-09T13:54:56.000+00:00

and then the function runs succesfully (lots of traces from out function).

But there is no trace of the first run (I would expect DequeueCount: 1, with the same InsertedOn) and a reason why it failed.

Is it possible that the host is just killed after that "CPU threshold exceeded" warning?

Why is the second try starting more than 10 minutes after queue insertion? My visibilityTimeout is 5 seconds.

Answer 1

"When a message is dequeued, it becomes invisible to other consumers for the duration of the visibilityTimeout."

It seems there is an internal visibilityTimeout set to 10 minutes, that is not explained in the documentation. What we, the users, can set is an additional time span. So, even if I set it to 0, we will have the second try after 10 minutes.

"Also, it is worth noting that the newBatchThreshold setting in host.json is set to 0, which means that the function will retrieve a new batch of messages as soon as the current batch is processed."

How else should I set it if I want no parallel processing?

From the documentation: "The maximum number of messages processed in parallel by the job is NewBatchThreshold plus BatchSize."

Answer 2

Hello @Anonymous Based on the provided information, it seems that the host was killed due to the CPU threshold being exceeded. This could have caused the first run to fail without leaving any trace. Regarding the delay before the second try, it is possible that the visibilityTimeout of 5 seconds was not enough for the function to complete processing the message.

When a message is dequeued, it becomes invisible to other consumers for the duration of the visibilityTimeout.

If the function takes longer than the visibilityTimeout to process the message, the message becomes visible again and can be dequeued by another consumer. In your case, it seems that the function took more than 5 seconds to process the message, causing it to become visible again and dequeued by the same function instance after more than 10 minutes.

You may want to consider increasing the visibilityTimeout to allow more time for the function to process the message.

Also, it is worth noting that the newBatchThreshold setting in host.json is set to 0, which means that the function will retrieve a new batch of messages as soon as the current batch is processed.

This could potentially cause the function to retrieve messages faster than it can process them, leading to delays and potential failures. You may want to consider increasing the newBatchThreshold to avoid this issue.