Azure Durable Functions timeout error in Activity Function

Chintan Rajvir 426 Reputation points Microsoft Employee
2021-03-24T05:36:56.127+00:00

I am running an Azure Durable Function in Python 3 runtime with 3 components: A HTTP Starter Function, An Orchestrator Function and an Activity Function. I am using "Consumption Plan SKU" of the App Service. In the execution trigger to the HTTP starter function, it will request the orchestrator to run about 250 instances of the activity function, each doing some work parallely. The orchestrator function snippet looks like:

   def orchestrator_function(context: df.DurableOrchestrationContext):  
       job_input = context.get_input()  
     
       parallel_tasks = []  
       for i in range(1, job_input["jobs"]+1):  
           payload = "<jsonPayload>"  
           parallel_tasks.append(context.call_activity("<activityFunctionName>", payload))  
     
       print("Scheduled jobs using durable activity function.")  
       results = yield context.task_all(parallel_tasks)  
         
       return results  

As I am doing a stress test with minimal compute, I have disabled the autoscale and currently there is only 1 host in the plan. I face an error of timeout in this scenario, where the error message suggests:

   Orchestrator function '<orchestratorFunctionName>' failed: Activity function '<activityFunctionName>' failed: Timeout value of 00:30:00 was exceeded by function: Functions.<activityFunctionName> \n {\"$type\":\"Microsoft.Azure.WebJobs.Host.FunctionTimeoutException  
   ...  

Using this link I can identify that the default timeout for "Consumption Plan" is 5 minutes. With my host.json not having any configuration done for the functionTimeout property, I am confused why is the error suggesting a timeout after 30 minutes? Please correct me if the above error is pointing to a different property or if the documentation needs to be modified. And what could be the optimal way to resolve this issue?

UPDATE:
To try out, I implemented the same scenario (orchestrator creating 250 activity calls), with functionTimeout set as "01:00:00". In the first try of this scenario, the app ended with the same error message as above at the end of 1 hour. In the second try of this scenario, the application is still running (above 2 hours) without any error prompt. The status query URI returns runtimeStatus as Running.

Azure Functions
Azure Functions
An Azure service that provides an event-driven serverless compute platform.
4,323 questions
{count} votes

Accepted answer
  1. JayaC-MSFT 5,526 Reputation points
    2021-04-01T16:52:12.593+00:00

    Hello @Chintan Rajvir , thanks for sharing the details shared by the Microsoft internal team. I am posting it here so that it helps others in the community:

    The timeout issue was because Linux hosts are not configured with default timeouts. There is a GitHub issue for the same : https://github.com/Azure/azure-functions-host/issues/5902

    For the second scenario ( there are exceptions of timeout occurring on the jobs, the query status function is still reporting the tasks to be running), Microsoft team is investigating the root cause. We'll will update the thread , if its not application specific.

    0 comments No comments

0 additional answers

Sort by: Most helpful