Durable Function delay (Function app in Consumption Plan)

Question

Hello,

I have a durable function (orchestrator function + 3 activity function) triggered by a regular function)
For some reason I observe unexpected long delay in the execution of the 'pipeline' (cf screenshot).
I tend to observe around 5/10 minutes in between execution of the different part of the orchestration (so not always between OrchestratorAnalysisStarter and OrchestratorActivityPrepare)
In ApplicationInsights I see no exception. So I just don't understand what's happening. So I don't think there's a user code error. It's just infrastructure latency. But it's quite heavy. Just wanted to know if you could point me in the right investigation direction.

Answer 1

Hi @MuthuKumaranMurugaachari-MSFT :
So the case is under investigation by MSFT-Support. So far we've come to the conclusion that the cause of the delay comes down to the fact that :

The lease for control queue is held by previous function instance which is supposed to be recycled

From time to time I observe 10' more delay.

When durable function has multiple instances, in order for these instances to cooperate with each other, durable function use Blob Lease as lock to ensure each control queue to be only processed by 1 instance.

In the following case (from few days ago), the issue happened because the previous instance “App-E7B79F19-638085047260461546” was instructed to be scaled in at 2023-01-05T09:15, but it was not actually removed until 2023-01-05T09:25

During the time, the function host on this instance had already stopped due to scale-in, so it didn’t actually execute any functions. In the meanwhile, it was still holding the lease of “eventtriggeredfunctions-control-03” queue, so the new instance “App-E7B79F19-638085066860627818” was not able to get the message from control queue either, then delay happened.

To the best of my knowledge i'm using the latest version of azure-functions-durable.

So my understanding is that it's a different case than issue : #2205 or #606