expired messages not accessible but still show in approximage message counts

Question

So I'm doing what you would think is very basic. Add messages to a queue manually in the web UI to validate basic processing/message hanlder behavior.
I'm using golang with github.com/Azure/azure-storage-queue-go/azqueue
I have a single storage queue and am not consuming messages from the client yet just ensuring all the counts/gets behave as expected as messages appear and disappear. This is in anticipation of I will have multiple instances of the client running that they handle the looks correctly before I bother processing/removing programmatically. And all the queue depth information will eventually be used for scaling automation.

The problem came in when I decided to validate with "expirations". So again just add/delete messages in the Azure portal. I then add a message that expires in 7 seconds. And DON'T delete it. The message properly disappears from the portal. Problem is the ApproximateMessagesCount (which in turn hits the REST api and this header: x-ms-approximate-messages-count:int-value) response count always includes the expired messages as part of the approximate-message-count. It states it "may" be more than actual. But it should eventually come out correct if no messages are moving in/out of the queue.

I assumed this may just take a while to clear - I've waited over an hour. I had max 5 messages and only 1 expired. It shouldn't take that long if there is a delay.
I thought maybe they hit the dead letter queue - Not that I can find (I can't even find that an auto dead letter exists for these queues only the service bus queues)

The ONLY ways I can find to "fix" the issue were
storage explorer (preview) - "clear queue" (not available anywhere else I can find including the Queues view in Storage Accounts)
delete queue (wait) recreate the queue

As far as I can tell this is not a new issue either. I found this is my digging. https://github.com/Azure/azure-storage-net/issues/645

Does anyone have anymore information on how to make this work correctly?

thanks

Answer 1

@voteplatypus Welcome to Microsoft Q&A, Thank you for posting your query here! Thanks for raising this question! Firstly, apologies for the delay in responding here and any inconvenience this issue may have caused. Kindly try the following steps to isolate the issue:

Valid values that can be entered in the Expires in field are between 1 second and 7 days. Select Message never expires to indicate a message that will remain in the queue until it is explicitly removed.

• Let us know how you are measuring the message count?

Approximate messages count will give you an approximate count of total messages in a queue and will include both visible and invisible messages.

One thing you could possibly do is PEEK at messages and it will return you a list of visible messages. However it will only return you a maximum of top 32 messages from the queue. So you logic to send notification message would work if the threshold is less than 32.

if you want to use ApproximateMessageCount to get result try this

CloudQueue q = queueClient.GetQueueReference(QUEUE_NAME);
q.FetchAttributes();
qCnt = q.ApproximateMessageCount;

Additional information: queue metrics are aggregated on service level other than queue level today. It’s in our roadmap but there is no concrete release plan yet.

The workaround is to run a reoccurring job to query the status of each queue. For example, query approximate message with Get Queue Metadata (REST API) - Azure Storage | Microsoft Learn.

Kindly let us know if you still have more questions on this., I wish to engage with you offline for a closer look and provide a quick and specialized assistance, please send an email with subject line “Attn:subm” to AzCommunity[at]Microsoft[dot]com referencing this thread and the Azure subscription ID, I will follow-up with you. Once again, apologies for any inconvenience with this issue.

Thanks for your patience and co-operation.

Hope this helps!

Kindly let us know if the above helps or you need further assistance on this issue.

-----------------------------------------------------------------------------------------------------------------------------------------------------------

Please don’t forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.

Answer 2

Hello

So I'm aware of peek and such as well I'm using all these things. I was just trying to dissect what appears to be a misbehavior/bug of the api.

So using golang.

Take

// Assume this pointer is connected properly already
queueUrl *azqueue.QueueURL 
props, _ := queueUrl.GetProperties(ctx)
messageCount := props.ApproximateMessagesCount()  // this reads header: x-ms-approximate-messages-count:int-value)

This will initially return 0 records on a brand new queue. Also I only have the One queue when testing this. No other queues in target subscription/resource group.
Then manipulate the Queue ONLY in the Azure Portal
Add a message (doesn't matter the contents) and run the above. Here's the flow

Add message (expire 7 days) (in portal)
messageCount == 1
Add message (expire 7 days)
messageCount == 2
Delete message (doesn't matter which)
messageCount == 1
Add message (expire 7 **seconds**)
messageCount == 2
wait about 10 seconds (in portal the 7sec expire message disappears)
messageCount == 2
Delete remaining visible message
messageCount == 1

And it stays that way forever more as far as I can tell.

thanks

Answer 3

@Sumarigo-MSFT

Could you share with us if there is any further update on this ? We want to use queue length for a scaling decision and hence need it to be reliable.

Answer 4

Hello Blue
I moved out of Azure to AWS due to it's excessive costs and bugs like this. They did contact me directly to "work through it" but if it can't be worked through on the public forum (so others can benefit as well) then I don't have time for it. Also the above code example showed the error very clearly. So I don't think they tried to reproduce.

I have not bothered testing since moving off Azure. But since it's been a bug for so long (issue linked in my first post above is from 2018) I'd assume it's still there. You can test using the above code flow easily though.

So I'd test before committing to it, but I'm also assuming it's still a bug. Oh and Peek won't work for scaling decisions in my opinion. As I recall it can only peek a max of 30? So if you have 3M messages you only know there are 30 more.

good luck!

Answer 5

Thank you for the detailed update @voteplatypus
From my analysis i am also disappointed at such a basic feature out of place, but i see no choice, but to avoid depending on it too.

We will still continue to use storage queues, but making sure our messages don't expire (high ttl), so that we don't get incorrect queue length counts, since our scaling logic does depend on it.