"Invocation cancelled - exiting retry loop." occurs on Function App Retry Mode

Jeph Bayfield 0 Reputation points
2023-11-22T13:18:44.3166667+00:00

I've experienced a few incidents of an azure function app shutting down in the past month. The function app is set configured to retry ("FixedDelayRetry") every 2 minutes for 59 times, until it runs successfully. This is required as a file is released at a random time within a 2 hour window and we need to pick it up as without too much delay.

This has worked great for a number of months now until recently. The logs are showing the following pattern:

  • DrainMode mode enabled
  • Calling StopAsync on the registered listeners
  • Stopping the listener 'Microsoft.Azure.WebJobs.Extensions.Http.HttpTriggerAttributeBindingProvider...
  • Stopped the listener 'Microsoft.Azure.WebJobs.Extensions.Http.HttpTriggerAttributeBindingProvider...
  • Stopping the listener 'Microsoft.Azure.WebJobs.Host.Listeners.SingletonListener...
  • Stopped the listener 'Microsoft.Azure.WebJobs.Host.Listeners.SingletonListener...
  • Invocation cancelled - exiting retry loop.

Is this caused by Drain mode kicking in? It states here that Drain mode should kick in at times during less load, however this seems a strange time to decide to do this, while the application is retrying a failed process after a 2/3 of the 59 attempts. Please could you give me an explanation for this behavior?

Azure Functions
Azure Functions
An Azure service that provides an event-driven serverless compute platform.
4,609 questions
{count} votes

1 answer

Sort by: Most helpful
  1. Mike Urnun 9,786 Reputation points Microsoft Employee
    2023-11-29T22:17:01.9133333+00:00

    Hello @Jeph Bayfield - Sorry for the late reply. This is a behavior that's been previously reported. The underlying issue affecting the TimeTrigger may not be related to the Drain Mode process at all, and it'll likely require deeper investigation with more details for troubleshooting the root cause. As such, we request that you try out the self-help steps described in the following resources first:

    If none of the steps work and you continue to observe the behavior where TimeTrigger functions aren't firing per schedule, feel free to work with us in a 1:1 setting via a support case route. If your subscription doesn't carry a support plan, let me know and I'll help open one for resolving this issue.

    related: https://learn.microsoft.com/en-us/answers/questions/770912/why-do-i-see-drainmode-mode-enabled-in-the-logs-an

    0 comments No comments