question

chintanr11-MSFT avatar image
0 Votes"
chintanr11-MSFT asked JayaC-MSFT answered

Azure Durable Functions timeout error in Activity Function

I am running an Azure Durable Function in Python 3 runtime with 3 components: A HTTP Starter Function, An Orchestrator Function and an Activity Function. I am using "Consumption Plan SKU" of the App Service. In the execution trigger to the HTTP starter function, it will request the orchestrator to run about 250 instances of the activity function, each doing some work parallely. The orchestrator function snippet looks like:

def orchestrator_function(context: df.DurableOrchestrationContext):
    job_input = context.get_input()

    parallel_tasks = []
    for i in range(1, job_input["jobs"]+1):
        payload = "<jsonPayload>"
        parallel_tasks.append(context.call_activity("<activityFunctionName>", payload))

    print("Scheduled jobs using durable activity function.")
    results = yield context.task_all(parallel_tasks)
    
    return results


As I am doing a stress test with minimal compute, I have disabled the autoscale and currently there is only 1 host in the plan. I face an error of timeout in this scenario, where the error message suggests:

Orchestrator function '<orchestratorFunctionName>' failed: Activity function '<activityFunctionName>' failed: Timeout value of 00:30:00 was exceeded by function: Functions.<activityFunctionName> \n {\"$type\":\"Microsoft.Azure.WebJobs.Host.FunctionTimeoutException
...


Using this link I can identify that the default timeout for "Consumption Plan" is 5 minutes. With my host.json not having any configuration done for the functionTimeout property, I am confused why is the error suggesting a timeout after 30 minutes? Please correct me if the above error is pointing to a different property or if the documentation needs to be modified. And what could be the optimal way to resolve this issue?

UPDATE:
To try out, I implemented the same scenario (orchestrator creating 250 activity calls), with functionTimeout set as "01:00:00". In the first try of this scenario, the app ended with the same error message as above at the end of 1 hour. In the second try of this scenario, the application is still running (above 2 hours) without any error prompt. The status query URI returns runtimeStatus as Running.



azure-functions
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@chintanr11-MSFT You are right about the Timeout Limits. Consumption Tier will not allow functions to run beyond 10 mins (max configurable). Based on the numbers, it is likely you are running on the Premium Tier. Could you double check?

The second scenario does seem off. The Premium Tier functions an run unbounded but guarantee only 60 minutes of runtime (sometimes they may be stopped at 60 minutes but at other times, could run for a very long time).


0 Votes 0 ·

I double-checked the plan. It shows consumption plan. I feel there is some mismatch between the document and the functioning. Also, with the consumption plan, I am able to observe higher timeouts by setting the functionTimeout property to 1 hour in the host.json. This leads us to the second scenario, where even after there are exceptions of timeout occurring on the jobs, the query status function is still reporting the tasks to be running.

0 Votes 0 ·

1 Answer

JayaC-MSFT avatar image
0 Votes"
JayaC-MSFT answered

Hello @chintanr11-MSFT, thanks for sharing the details shared by the Microsoft internal team. I am posting it here so that it helps others in the community:

The timeout issue was because Linux hosts are not configured with default timeouts. There is a GitHub issue for the same : https://github.com/Azure/azure-functions-host/issues/5902

For the second scenario ( there are exceptions of timeout occurring on the jobs, the query status function is still reporting the tasks to be running), Microsoft team is investigating the root cause. We'll will update the thread , if its not application specific.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.