Azure Storage Queue: After visibility time out of message, message appears at the end of the queue

Question

I need to retrieve one message from the front of queue & process it. If for any reason message is not processed the message should reappear on the front of the queue as 1st message, after visibility timeout.
However I am observing a different behavior in storage queue, where once a message is retrieved & after visibility timeout it reappears at the end of the queue. Is is expected.

Answer 1

@Abinash Kumar Dalai Thank you for reaching out to Microsoft Q&A. I understand that you are having issues with Azure Storage Queue where after the Visibility time out expires, the message appears at the end of the queue instead of the top of the queue.

Azure Storage Queue does not provide Ordering guarantees. However, Storage bus queue is able to provide First In First Out (FIFO) capabilities by using message sessions.

Please check this document that compares both the queues: https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-azure-and-service-bus-queues-compared-contrasted

• Messages in Storage Queues are typically first-in-first-out, but sometimes they can be out of order. For example, when the visibility-timeout duration of a message expires because a client application crashed while processing a message. When the visibility timeout expires, the message becomes visible again on the queue for another worker to dequeue it. At that point, the newly visible message might be placed in the queue to be dequeued again.

Hope this helps. Please let us know if you have any further questions and we will be glad to assist you further. Thank you!

Remember:

Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.

Want a reminder to come back and check responses? Here is how to subscribe to a notification.

Answer 2

@SaiKishor-MSFT Thanks for the reply. I observe some ambiguity (lack of clarity) over the documents. If you can please clarify considering the below statements , it will be much helpful -

Consider this scenario, 5 messages are there in queue at sequential position 1 to 5 (A,B,C,D,E) & a GET operation is performed on the message A with visibilitytimeout as 20secs, however it is not deleted. So after 20secs is elapsed, where the message should re-appear - at it's original position or the message B is now front of the queue & A moves to position 5.

• the message becomes visible again on the queue for another worker to dequeue it. At that point, the newly visible message might be placed in the queue to be dequeued again..* - This statement is fairly confusing, where the message will be placed in it's original position or some arbitrary position.

Answer 3

@Abinash Kumar Dalai Thank you for your patience while I was looking into this. Please find more details regarding your question below-

Short answer: Application crashes while processing a message, lead to visibility timeout expiry and the order of messages queued will differ based on whether there are other messages with more visibility timeout.

Long answer (with a scenario):

Ideally, assuming all the messages are issued “get” command with a same visibility timeout, it will be added fifo as usual as shown below.

Below case, I put hello1 to hello5 and issued 2 gets.

Message put back to the end of queue with an updated dequeue count in the same order, hello1 and hello2. Next time, I issue get, hello3 will be the next in position to be dequeued and then hello4, hello5, hello1, hello2 and so on..

Now, let us assume, 2 different client applications issue 2 gets and 2 messages are dequeued of the queue. If the 2nd application crashes while processing the message, its visibility timeout expires but 1st message’s visibility timeout might still be there. In that case, that message will be put back in the order of 2nd message and then the 1st.

For testing this, I purposefully added a smaller timeout (mimicking a crash).

As you see, I popped hello3 first and hello4 next but hello4 is queued before hello3. This is what would happen in case of application crash/visibility timeout expiry.

Hope this helps. Please let us know if you have any further questions and we will be glad to assist you further. Thank you!

Remember:

Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.

Want a reminder to come back and check responses? Here is how to subscribe to a notification.