Azure Storage Queue deletes messages

Question

I have a WebJob running two different Quartz.net jobs. They both read from the Azure Queue the same way, but one of the jobs keeps loosing messages, I feel like I have tried everything but I cannot figure out what is wrong.

Here are the steps

1. Open the Queue
2. Get the Properties and read the ApproximateMessagesCount
3. Call ReceiveMessagesAsync with the count found from above
4. Sometimes I then get my messages, some times only some of them but most times I receive none

Here is an example from my log

[INFO] Approximate message count is 1 in '{queuename}'
[WARN] [ReceiveMessagesAsync] Asked for 1 messages. Received 0 messages from '{queuename}'

I can see in the Storage Explorer that the message is there, but after ReceiveMessagesAsync has been called it's gone. Not hidden, but gone.

I can see that ReceiveMessagesAsync receives an empty xml list from Azure, so somehow calling ReceiveMessagesAsync deletes my messages.
I cannot replicate this locally, only when running the code as a WebJob and theres is no pattern in when it decides to delete my messages

I am running .net core 6 using the Azure.Storage.Queues version 12.22.0 nuget package

Answer 1

hi Michael!

thanks for posting this, it's a tricky one for sure ))

when u call receivemessagesasync, azure queue storage actually does two things at once: it gets the messages AND makes them invisible for a short time (by default 30 seconds). but in ur case, something's going wrong in this process )) let's check the basics. u're using the right approach with approximatecount, but there's a sneaky thing, this count isn't always 100% accurate. microsoft docs even say it's... well, approximate :) microsoft docs link

now about ur disappearing messages, this usually happens when

the visibility timeout expires before u process the message

someone else is also reading the same queue

there's some network hiccup during the operation

try this fix add a small delay after getting the count. sometimes the queue needs a moment to sync up. also, set a longer visibility timeout just to be safe.

var options = new queuereceivemessagesoptions 
{
    visibilitytimeout = timespan.fromminutes(5) // give urself more time
};

var messages = await queueclient.receivemessagesasync(maxmessages: count, options: options);

if that doesn't help, maybe add some retry logic. azure storage can sometimes be flaky, especially under heavy load. microsoft suggests using their built-in retry policies, retry policy docs

one more thing, check if u have multiple instances of ur webjob running. if two instances grab the same message at nearly the same time... poof! message disappears :))

try these tweaks and let me know how it goes %)

Best regards,

Alex

and "yes" if you would follow me at Q&A - personaly thx.
P.S. If my answer help to you, please Accept my answer
PPS That is my Answer and not a Comment

https://ctrlaltdel.blog/

Answer 2

Here is an example:

3 messages was inserted

ApproximateMessagesCount returned 3

ReceiveMessagesAsync was called with a maxMessages of 3

No messages was returned and now the queue is empty. So I lost all my messages.

The content is a simple json string with one property, I have tried using base64 encoding and not encoding with the same result.