Azure functions not processing all the messages

Niraj 45

Hi, I have this issue where the azure function is not processing all the messages. I confirmed in the service bus if all my messages were there or not and I could see all my 150 messages in the queue in the azure portal. When the function gets triggered it doesnt process all the messages. The session id is same for all messages. Also in application properties of the message I send this json
{'msgCount': 1, 'msgTotal': 150, 'isLastMessage': False, 'isFirstMessage': True}
{'msgCount': 2, 'msgTotal': 150, 'isLastMessage': False, 'isFirstMessage': False}

................

{'msgCount': 150, 'msgTotal': 150, 'isLastMessage': True, 'isFirstMessage': False}.

What happens is I get the messages in squence but some of the messages in between get dropped.
Need help been facing this issue since last 3 months

Konstantinos Passadis 19,171 Reputation points MVP

2023-07-24T10:03:29.1766667+00:00

Hello @Niraj !

Welcome to Microsoft QnA!

based on your input the problem seems to be in Azure Functions integration with Service Bus

Please have a look for

exceptions during Azure Functions Executions , so check Application Insights and Functions Logging

Azure Functions have a maximum timeout limit (5 minutes by default for Consumption plan, which can be increased up to 10 minutes). If your function is taking more than the timeout limit to process a single message, it will be terminated, and the message will be considered as not processed.

Service Bus Lock Duration :

The maximum value is 5 minutes, so you need to renew the lock if you want to have this longer. Having a longer lock duration than needed has some implications as well. For example, when your client stops working, the message will only become available again after the lock duration has passed

https://learn.microsoft.com/en-us/azure/service-bus-messaging/message-transfers-locks-settlement

Also :

https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-performance-improvements?tabs=net-standard-sdk-2

https://learn.microsoft.com/en-us/azure/service-bus-messaging/service-bus-prefetch?tabs=dotnet

When you enable the Prefetch feature for any of the official Service Bus clients, the receiver acquires more messages than what the application initially asked for, up to the specified prefetch count. As messages are returned to the application, the client acquires further messages in the background, to fill the prefetch buffer.

You could also share any errors seen in Azure Functions Logs , but check the info provided beforehands

Kindly let us know !

I hope this helps!

Kindly mark the answer as Accepted and Upvote in case it helped!

Regards
Niraj 45 Reputation points

2023-07-24T10:39:04.0933333+00:00

I think so I fixed it by adding the message_id to the message which I earlier was not passing. Will run it few more times to confirm if its working but now getting all the messages. This was never happening earlier
Konstantinos Passadis 19,171 Reputation points MVP

2023-07-24T14:20:21.9533333+00:00

Hello @Niraj !

Thats great ! i hope it works as it should !

In case my answer helped kindly mark it as Accepted and Upvote !

Thank you !
Niraj 45 Reputation points

2023-07-24T14:44:45.8033333+00:00

thank you
Niraj 45 Reputation points

2024-01-31T05:08:27.7266667+00:00

@Konstantinos Passadis @MayankBargali-MSFT
Hey, I would like to open this conversation again. I thought that by adding message_id to ServiceBusMessage would fix it but that is still not happening. I send N number of messages to the service bus queue. I have a service bus queue trigger function listening to this. The issue is the function doesnt process all the messages. It skips the messages in between. Now How do I know this is because I set a number to every message in the message_properties like 1,2,3,5 and so on till the last message. In the function app log stream I print these numbers for example in the images below. In the log stream I dont see all the messages being processed and no error for the messages which are not getting processed
MayankBargali-MSFT 70,531 Reputation points

2024-01-31T10:20:58.4766667+00:00

@Niraj I can sync with you offline to review it. Please send email with subject line “Attn:Mayank” to AzCommunity[at]Microsoft[dot]com along with your availability in UTC.Edited: Looks like there was issue in posting the comment.
Niraj 45 Reputation points

2024-01-31T10:50:08.55+00:00

@MayankBargali-MSFT how can we connect?
pardeep kumar 0 Reputation points

2024-07-08T05:37:22.6533333+00:00

Please provide the solution. I am facing same issue.

Accepted answer

MayankBargali-MSFT 70,531 Reputation points

2023-07-25T05:36:28.8133333+00:00

Hi @Niraj

I'm glad to see you were able to resolve your issue. Thanks for posting your solution so that others experiencing the same thing can easily reference this.

Since the Microsoft Q&A community has a policy that the question author cannot accept their own answer, they can only accept answers by others, I'll repost your solution in case you'd like to Accept the answer.

Issue: You observe that your azure function was not processing all the messages. You are using the same session id for all your messages. You observe that the messages are not getting in sequence and looks like some messages are dropped.

Resolve: You have updated your code and added the message id to the message which was previously not passed.
Please sign in to rate this answer.

1 person found this answer helpful.

0 comments No comments
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Azure functions not processing all the messages

0 additional answers

Your answer