Service Bus Timeout Exception

Question

We have a service bus topic with 300+ number of subscriptions, all these subscribers are azure functions (python) and we are facing a timeout exception that we are trying to understand better and is where we would appreciate inputs from the community please.

 

Currently the timeout is set to 5 minutes and we are on consumption plan.

 

Sharing below the configuration details for function app (host.json) along with the call stack for the exception. Any help on how to counter this would be appreciated.

 

{
    "version": "2.0",
    "extensions": {
        "serviceBus": {
            "batchOptions": {
                "maxMessageCount": 10000,
                "operationTimeout": "00:05:00",
                "autoComplete": true
            },
            "prefetchCount": 0
        }
    },
    "logging": {
        "fileLoggingMode": "always",
        "logLevel": {
            "default": "Information",
            "Host.Results": "Information",
            "Function": "Information",
            "Host.Aggregator": "Information"
        },
        "applicationInsights": {
            "samplingSettings": {
                "isEnabled": false
            }
        }
    },
    "extensionBundle": {
        "id": "Microsoft.Azure.Functions.ExtensionBundle",
        "version": "[3.*, 4.0.0)"
    },
    "concurrency": {
        "dynamicConcurrencyEnabled": true,
        "snapshotPersistenceEnabled": true
    }
}

 

Microsoft.Azure.WebJobs.Host.FunctionTimeoutException:
   at Microsoft.Azure.WebJobs.ServiceBus.MessageProcessor.CompleteProcessingMessageAsync (Microsoft.Azure.WebJobs.Extensions.ServiceBus, Version=5.8.1.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at Microsoft.Azure.WebJobs.ServiceBus.Listeners.ServiceBusListener+<ProcessMessageAsync>d__27.MoveNext (Microsoft.Azure.WebJobs.Extensions.ServiceBus, Version=5.8.1.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable+ConfiguredTaskAwaiter.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Azure.Messaging.ServiceBus.ServiceBusProcessor+<OnProcessMessageAsync>d__104.MoveNext (Azure.Messaging.ServiceBus, Version=7.11.1.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable+ConfiguredTaskAwaiter.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Azure.Messaging.ServiceBus.ReceiverManager+<OnMessageHandler>d__19.MoveNext (Azure.Messaging.ServiceBus, Version=7.11.1.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable+ConfiguredTaskAwaiter.GetResult (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Azure.Messaging.ServiceBus.ReceiverManager+<ProcessOneMessage>d__15.MoveNext (Azure.Messaging.ServiceBus, Version=7.11.1.0, Culture=neutral, PublicKeyToken=92742159e12e44c8)

Answer 1

@Anonymous Thanks for reaching out. As per the error message it looks like the function is getting timeout while executing. As you are using the consumption function app, so the default timeout of your function app is 5 min and max 10 minutes as documented here. As per the configuration I can see you have specified 1000 for the maxMessageCount and that means 1000 messages will be sent to the function when triggered. So, it looks like your function may not be able to process these many messages or it is taking time to process as per your business logic code. The suggestion would be to decrease this value to a low value let's say 10 and then check what the average time is to execute your function and then set it accordingly.

You can always review Diagnose and solve problems blade on your function app for any issue and recommended steps to resolve the issue.

Feel free to get back to me if you need any assistance.

Please accept as "Yes" if the answer is helpful so that it can help others in the community. If you need any help/clarification/concerns/follow up questions, then please click on "Add Comment" on my answer and provide more details.