Python Durable Function Delayed Processing

Question

I have a durable function built using Python. At a high level, it takes as input a message with a list of objects. The orchestrator does a fan-out/fan-in by calling a sub-orchestrator. The sub-orchestrator then does function chaining with several activity functions. This setup is working well for the most part. However, I have noticed some issues when under moderate load. I queued up 50 messages to the function. 48 of them went through no problem in just a couple minutes. But a few seemed to get stuck.

Looking at the history and logs, what I have seen is that there seems to be a long pause between 2 of the activity functions called by the sub-orchestrator for no apparent reason. All of the other messages had completed fully by 8:47, however, these last few didn't resume until 8:53. It can't be resource issue because nothing was processing during this gap. This activity is not being called with a retry policy either. What might be causing this long delay? I can also attach code if that would be helpful.

Here is a screenshot of the Gantt Chart. As you can see, there is a large gap between save_results and save_metrics activities for one instance of the sub-orchestration. However the actual processing time of the activities is tiny.

Answer 1

Hi Phillip S. Stenger Jr,

The delays you are seeing where the system idles before launching the second-to-last or last activity are documented behaviors in Azure Durable Functions. This happens because the orchestrator is simply "waiting for work" and not actually executing any activities during those idle gaps.

 

Queue Polling and Control Queue Lease

·         The Durable Functions orchestrator uses Azure Storage control queues to schedule next steps. When nothing is running, the orchestrator instance waits for a new control message indicating the next activity should run.

·         If the previous activity was very fast or the instance is scaling out/in, sometimes there’s a short period before the next message is dequeued and processed.

·         The control queue is polled with exponential back-off, so after several idle checks, the wait between polls can reach up to 30 seconds by default. This is the primary reason for unexpected idle gaps.

 

Reference

Durable Functions Troubleshooting Guide - Azure Functions | Azure Docs

Azure Storage provider for Durable Functions | Microsoft Learn

Performance issues using Durable Entities · Issue #1447 · Azure/azure-functions-durable-extension

 

Kindly let us know if the above helps or you need further assistance on this issue.

Please "Upvote" if the information helped you. This will help us and others in the community as well.

Thanks