Durable function orchestrator stuck in pending

Question

Durable function orchestrator stuck in pending

Kevin Vanthuyne 0

We are experiencing an issue with our Azure Durable Functions App. When triggering a new run through an HTTP trigger, the status stays on PENDING.

Context

Some more context: we are using Linux, Python and the Flex Consumption plan. Our function app consists out of an HTTP trigger, an orchestrator and 4 activity functions.

We can see the HTTP trigger getting triggered correctly through the Invocations tab. The orchestrator does not show any invocations, but does show orchestrations, although stuck on the Pending runtime status.

The activity log also shows failed "List" operations, with the following (redacted) summary:

Resource: /subscriptions/X/resourceGroups/X/providers/Microsoft.Web/sites/X/functions/http_trigger/config/appsettings
Operation name: List
Time stamp: Mon Apr 21 2025 09:38:01 GMT+0200 (Central European Summer Time)
Event initiated by: X
Error: codeBadRequest
Message: {"error":{"code":"BadRequest","message":""}}

As far as we know, these are all the symptoms we can observe.

Tried solutions

We have restarted the function app, redeployed it, and terminated the orchestration instance through its API, to no avail.

Presumed issues

We think the functions queue might be in an erroneous state, causing orchestrations not to continue.

Possible cause

We think this might have been caused by deploying function code with an error in it. After executing the function, the orchestrator completed with the "Failed" state, and ever since, orchestrations have been stuck on pending. We have a production and development environment, and this happened for both environments.

Question

What is going wrong here and how can we fix this?

Sai Prabhu Naveen Parimi 2,420 Reputation points Microsoft External Staff Moderator

2025-04-21T08:10:48.4366667+00:00
Hi @Kevin Vanthuyne

We are currently investigating the issue with the orchestrator in our Azure Durable Functions app, where it is stuck in a "Pending" state. Based on the error provided, we wanted to ask a few questions to better understand the root cause:

App Settings Configuration: Could you please confirm whether the app settings (such as AzureWebJobsStorage) are configured correctly for the function app?

Permissions and Access: Is the function app able to access all the required resources, like the storage account or queues, without any permission issues?

Deployment Review: Was there any issue during the most recent deployment that might have caused a misconfiguration in the appsettings?

Azure Permissions: Could there be any permission issues preventing access or modification of the function app’s settings, as indicated by the BadRequest error?

Reference : https://learn.midurable/durable-functions-troubleshooting-guide#orchestration-is-stuck-in-the-pending-state

Your insights into these questions would help us narrow down the cause and resolve the issue more efficiently.
Kevin Vanthuyne 0 Reputation points

2025-04-21T08:45:10.2533333+00:00
This error started occurring without any changes to our working infrastructure.

I double checked, and the environment variables like AzureWebJobsStorage are still configured like before.

When checking the Access Control (IAM) tab of the storage account that belongs to the functions app, I don't see any roles assigned to the functions app. Should there be any? The storage account was created automatically as part of creating the functions app, so I assume this should be correct? Also, like I said, this used to work before.

We deploy using GitHub Actions and the Azure/functions-action. I see no errors or warning in the logs of this action. I also tried deploying locally from VS Code but this does not make a difference.

I don't know how to verify this.
Sai Prabhu Naveen Parimi 2,420 Reputation points Microsoft External Staff Moderator

2025-04-22T09:39:18.12+00:00

@Kevin Vanthuyne

Could you please check the private message and provide the required details for troubleshooting the issue further.

Here is the reference link on how to access & data retention policy of private messages in Microsoft Q&A.
RithwikBojja 3,055 Reputation points Microsoft External Staff Moderator

2025-04-25T04:49:44.3466667+00:00

Hi @Kevin Vanthuyne,

I think that the main cause of the issue was likely due to the FunctionTimeout being set as Unlimited in the Dedicated App Service Plan. This configuration can cause Azure Durable Functions orchestrations triggered via HTTP to remain in a "Pending" state, as it may lead to the function being in a hang or deadlock status.

Solution Recommended:

We recommended setting the FunctionTimeout to a limited value (e.g., 30 minutes) in the host.json file. Additionally, we suggested manually restarting the function in the Azure portal to mitigate the issue quickly. We also advised increasing the partition count, as having fewer partitions than workers can limit the processing of orchestration messages.
RithwikBojja 3,055 Reputation points Microsoft External Staff Moderator

2025-04-28T04:05:48.9+00:00

Hi @Kevin Vanthuyne,, have you got any chance to check my reply?
Kevin Vanthuyne 0 Reputation points

2025-04-28T06:09:53.93+00:00

@RithwikBojja that's definitely an interesting option. It seems like a good idea to indeed set it to a limited amount of time instead of unlimited. However, your suggestion of restarting the function manually had no impact on the stuck orchestrator. We did this multiple times and even redeployed multiple times but this did not fix the pending orchestrator state.
Kevin Vanthuyne 0 Reputation points

2025-04-28T12:01:02.38+00:00

The issue seemed to have resolved itself out of the blue after spending 3-4 full days trying to restart the function, terminating all instances, redeploying the same code, deploying minimal examples, deploying another Azure Function App, when nothing seemed to work.

So not sure what caused this. Sai indicated the error I was seeing was due to an error in our code, but like I said, our development environment that was stuck in pending suddenly started working again without deploying any new code, so I don't think that was the issue.

As for Rithwik's suggestion, I cannot verify if the timeout would resolve the issue as currently we don't have any orchestrators stuck in pending anymore. However, restarting the function manually did not resolve the issue, so that is definitely not a solution.

Your answer

Sai Prabhu Naveen Parimi 2,420 Reputation points Microsoft External Staff Moderator

2025-04-21T08:10:48.4366667+00:00

Hi @Kevin Vanthuyne

We are currently investigating the issue with the orchestrator in our Azure Durable Functions app, where it is stuck in a "Pending" state. Based on the error provided, we wanted to ask a few questions to better understand the root cause:

App Settings Configuration: Could you please confirm whether the app settings (such as AzureWebJobsStorage) are configured correctly for the function app?

Permissions and Access: Is the function app able to access all the required resources, like the storage account or queues, without any permission issues?

Deployment Review: Was there any issue during the most recent deployment that might have caused a misconfiguration in the appsettings?

Azure Permissions: Could there be any permission issues preventing access or modification of the function app’s settings, as indicated by the BadRequest error?

Reference : https://learn.midurable/durable-functions-troubleshooting-guide#orchestration-is-stuck-in-the-pending-state

Your insights into these questions would help us narrow down the cause and resolve the issue more efficiently.
Kevin Vanthuyne 0 Reputation points

2025-04-21T08:45:10.2533333+00:00

This error started occurring without any changes to our working infrastructure.

I double checked, and the environment variables like AzureWebJobsStorage are still configured like before.

When checking the Access Control (IAM) tab of the storage account that belongs to the functions app, I don't see any roles assigned to the functions app. Should there be any? The storage account was created automatically as part of creating the functions app, so I assume this should be correct? Also, like I said, this used to work before.

We deploy using GitHub Actions and the Azure/functions-action. I see no errors or warning in the logs of this action. I also tried deploying locally from VS Code but this does not make a difference.

I don't know how to verify this.
Sai Prabhu Naveen Parimi 2,420 Reputation points Microsoft External Staff Moderator

2025-04-22T09:39:18.12+00:00

@Kevin Vanthuyne

Could you please check the private message and provide the required details for troubleshooting the issue further.

Here is the reference link on how to access & data retention policy of private messages in Microsoft Q&A.
RithwikBojja 3,055 Reputation points Microsoft External Staff Moderator

2025-04-25T04:49:44.3466667+00:00

Hi @Kevin Vanthuyne,

I think that the main cause of the issue was likely due to the FunctionTimeout being set as Unlimited in the Dedicated App Service Plan. This configuration can cause Azure Durable Functions orchestrations triggered via HTTP to remain in a "Pending" state, as it may lead to the function being in a hang or deadlock status.

Solution Recommended:

We recommended setting the FunctionTimeout to a limited value (e.g., 30 minutes) in the host.json file. Additionally, we suggested manually restarting the function in the Azure portal to mitigate the issue quickly. We also advised increasing the partition count, as having fewer partitions than workers can limit the processing of orchestration messages.
RithwikBojja 3,055 Reputation points Microsoft External Staff Moderator

2025-04-28T04:05:48.9+00:00

Hi @Kevin Vanthuyne,, have you got any chance to check my reply?
Kevin Vanthuyne 0 Reputation points

2025-04-28T06:09:53.93+00:00

@RithwikBojja that's definitely an interesting option. It seems like a good idea to indeed set it to a limited amount of time instead of unlimited. However, your suggestion of restarting the function manually had no impact on the stuck orchestrator. We did this multiple times and even redeployed multiple times but this did not fix the pending orchestrator state.
Kevin Vanthuyne 0 Reputation points

2025-04-28T12:01:02.38+00:00

The issue seemed to have resolved itself out of the blue after spending 3-4 full days trying to restart the function, terminating all instances, redeploying the same code, deploying minimal examples, deploying another Azure Function App, when nothing seemed to work.

So not sure what caused this. Sai indicated the error I was seeing was due to an error in our code, but like I said, our development environment that was stuck in pending suddenly started working again without deploying any new code, so I don't think that was the issue.

As for Rithwik's suggestion, I cannot verify if the timeout would resolve the issue as currently we don't have any orchestrators stuck in pending anymore. However, restarting the function manually did not resolve the issue, so that is definitely not a solution.

Share via

Durable function orchestrator stuck in pending

Your answer