question

PaulColeman-4426 avatar image
3 Votes"
PaulColeman-4426 asked LouisGordon-4914 published

dotnet-isolated ServiceBusTrigger very slow with IsSessionsEnabled

Hello,

I am seeing very slow ServiceBusTrigger trigger behavior for queues that are session enabled. It looks like a 'batch' amount of messages are being read from the queue, and the function called. However after the initial calls, nothing then happens until about 60s then another 'batch' amount are being called. The messages are being processed, just at a rate of about 10 per 60s.

I can reproduce this locally and deployed in an Azure function. This behavior doesn't happen when not using .net6/isolated functions. And also doesn't happen for non-session-enabled queues.

The code looks like this:

     [Function("DotNetIsolatedSessionsIssueTestQueue")]
     public async Task DotNetIsolatedSessionsIssueTestQueue(
         [ServiceBusTrigger("test1", Connection = "ServiceBusConnection", IsSessionsEnabled = true)] byte[] message,
         FunctionContext context)
     {
         await Task.CompletedTask;
     }

Is this a known problem? Any ideas on how to fix it?

There is nothing wild in my host.json file:
{
"version": "2.0"
}

Nor the local.settings.json:
{
"IsEncrypted": false,
"Values": {
"FUNCTIONS_WORKER_RUNTIME": "dotnet-isolated",
"FUNCTIONS_EXTENSION_VERSION": "~4",
"AzureWebJobsStorage": "UseDevelopmentStorage=true",
"ServiceBusConnection": "Endpoint=xx;SharedAccessKeyName=ServicesKey;SharedAccessKey=***",
}
}

azure-functionsazure-service-bus
· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

We are experiencing the same thing here after we updated from .net 3.1 to .net 6.
Altough our runtime is actually non-isolated

 "FUNCTIONS_WORKER_RUNTIME": "dotnet"

During the update, we also updated the Servicebus NuGet package, so we arent quite sure yet which of the changes that caused the sudden delay.

<PackageReference Include="Microsoft.Azure.ServiceBus" Version="5.2.0" />

As this causes some unwanted sideeffects, we havent yet promoted anything to our production, and as a result, we can see a clear difference in our queue backlog at the servicebus between our test environment and production.


0 Votes 0 ·

Thanks for including the info that it's not just dotnet-isolated! It's really strange.

I have a support case open with Microsoft on this issue too; if I make any progress on it with them I'll let you know. Plus I opened this: https://github.com/Azure/azure-functions-dotnet-worker/issues/831

0 Votes 0 ·

Did you hear anything back? I am also seeing this issue.

0 Votes 0 ·
MughundhanRaveendran-MSFT avatar image
0 Votes"
MughundhanRaveendran-MSFT answered PaulColeman-4426 commented

@PaulColeman-4426 ,

Welcome to Q&A and thanks for posting your query here.

First I would suggest you to make sure that the "Always On" option (available in configuration blade) is turned on so that cold starts are avoided. Please note that, this is applicable only for function apps running on dedicated app service plan and elastic premium sku.

Concurrency and batching is where performance can be improved from Functions platform perspective. It is upto you configure the number of messages that needs to be processed in a batch and you can have upto 16 concurrent sessions under the sessionhandler options. We can add sessionhandleroptions under the host.json file

"sessionHandlerOptions": {
"autoComplete": false,
"messageWaitTimeout": "00:00:30",
"maxAutoRenewDuration": "00:55:00",
"maxConcurrentSessions": 16
}
Host.json reference : https://docs.microsoft.com/en-us/azure/azure-functions/functions-bindings-service-bus#hostjson-settings

Please try this configuration and let me know the outcome.

I hope this helps!

Please 'Accept as answer' and ‘Upvote’ if it helped so that it can help others in the community looking for help on similar topics.

· 5
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

I don't think that's the answer. The function app is not cold because it processed the 'batch' of messages immediately before waiting. Here's a more generalized version of the behavior I'm seeing. Assume that the queue has enough messages in it that messages are always waiting to be processed:

[read x messages, process them okay] => wait 60 seconds => [read x messages, process them okay] => wait 60 seconds => ...

This keeps going forever. It isn't a cold start issue, or a batch size/concurrency issue as the waiting in between still happens. The messages are processed okay and well under the message time out, so that's not the issue either. The waiting isn't waiting for messages to arrive in the queue (because there are plenty in there), and the waiting isn't for messages to be processed because they get processed in under a second. The waiting seems to be polling or something similar; and this is only happening when using dotnet-isolated.

Any other ideas?

0 Votes 0 ·

@PaulColeman-4426 , I had 18k messages in my SB queue and the dotnet-isolated function app was able to process the messages in 13 minutes continously, I didnt see any wait time. I am just using the template code

[Function("Function1")]
public static void Run([ServiceBusTrigger("session-queue", Connection = "constring",IsSessionsEnabled =true)] string myQueueItem, FunctionContext context)
{
var logger = context.GetLogger("Function1");
logger.LogInformation($"C# ServiceBus queue trigger function processed message: {myQueueItem}");
}

Could you please try the template code and check if you see the issue or not?

0 Votes 0 ·
PaulColeman-4426 avatar image PaulColeman-4426 MughundhanRaveendran-MSFT ·

I just discovered that if I use the same session id for all the messages they are processed quickly. However if the messages have unique session ids is when the pausing between batches happens.

In your test above did you use the same session id for all messages or different?

0 Votes 0 ·
Show more comments
PaulColeman-4426 avatar image
0 Votes"
PaulColeman-4426 answered

It appears with version 5.x of the service bus SDK things changed:

"The previous illustration shows three concurrent session receivers. One Session with SessionId = 4 has no active, owning client, which means that no messages are delivered from this specific session. A session acts in many ways like a sub queue."

https://docs.microsoft.com/en-us/azure/service-bus-messaging/message-sessions#session-features

The default current streams is 8 apparently. So once my 8 streams have been 'used' by 8 unique session ids, the streams sit waiting for sessionIdleTimeout to look for a different session id to pick up.

This behavior I'm seeing is by (new) design then. To change sessionIdleTimeout you can do something like this in your hosts.json file:
{ "version": "2.0", "extensions": { "serviceBus": { "sessionIdleTimeout": "00:00:20" } } }

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.