How to gracefully terminate an eternal orchestration in Azure Durable Functions

Question

Hello,

I have an Azure Function in python that has an eternal orchestrator.

This eternal orchestrator checks for messages in a message queue and then processes it.

After the message has been processed it is written into an output-queue so that the system knows if the whole process was successful or not.

I now want to implement a way to gracefully stop the eternal orchestrator.

This means that all the messages that are currently being worked will be processed until they are written into the output-queue but it will no longer check for new messages.

How would I implement this using an HTTP-Trigger? I have read that you can use a .terminate to do something like this, however I don't know how I can save the orchestration_id for this and I also don't know if that is a graceful stop or a sudden one.

Any help would be appreciated.

Answer 1

Hello @Benedikt Schmitt,
To gracefully terminate an eternal orchestration in Azure Durable Functions, you can use the terminate API of the orchestration client binding. This allows you to stop the orchestration while ensuring that all currently processing messages are completed before the orchestration halts.

Here is a general approach to implement this using an HTTP trigger:

• Store the Orchestration ID: When you start your eternal orchestration, make sure to store the orchestration ID somewhere accessible, such as in a database or in-memory storage.
• Implement the HTTP Trigger: Create an HTTP-triggered function that will handle the termination request. This function should retrieve the stored orchestration ID.
• Call the Terminate API: Use the terminate method on the orchestration client to stop the orchestration. You can provide an optional reason for termination, but this is not necessary for a graceful stop.
• Graceful Processing: The orchestration will continue to process any messages it is currently handling until they are completed, but it will not check for new messages after the terminate call.

For further clarification, please refer to the following documentations:

• Eternal orchestrations in Durable Functions (Azure Functions)
• Handling errors in Durable Functions (Azure Functions)

I hope this helps resolve your issue. Feel free to reach out if you have further concerns.

Answer 2

Hello Benedikt Schmitt,

Welcome to the Microsoft Q&A and thank you for posting your questions here.

I understand that you need how you can gracefully terminate an eternal orchestration in Azure Durable Functions.

Most of all, @Ranashekar Guda have done a great job. However, the below are additional to @Ranashekar Guda and alternative method to do it:

Additional detailed steps with references:

1. When starting the eternal orchestration, store the orchestration ID in a database or in-memory storage for easy retrieval.

         orchestration_id = client.start_new('EternalOrchestrator', None)
         # Store orchestration_id in a database or in-memory storage
   

2. Create an HTTP-triggered function to handle termination requests.

         @app.route('/terminate', methods=['POST'])
         def terminate_orchestration():
             orchestration_id = retrieve_orchestration_id()  # Retrieve from storage
             client.terminate(orchestration_id, "Graceful stop")
             return "Termination initiated", 202
   

3. Use the terminate method on the orchestration client to stop the orchestration gracefully.

         client.terminate(orchestration_id, "Graceful stop")
   

4. The orchestration will continue processing current messages but will not check for new ones after the terminate call. Check these two links:
   • https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-eternal-orchestrations
   • https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-instance-management).

         async def EternalOrchestrator(context):
             while True:
                 message = await context.wait_for_external_event('NewMessage')
                 await context.call_activity('ProcessMessage', message)
                 await context.call_activity('WriteToOutputQueue', message)
                 if context.is_terminating():
                     break
   

Alternative methods:

Should there be any issue that the terminate API isn't working for gracefully terminating an eternal orchestration in Azure Durable Functions as expected, you can leverage on the below.

1. The ContinueAsNew method allows you to reset the state of the orchestration and start it anew with a new input. This can be used to gracefully stop the current orchestration and start a new one without processing new messages.

      async def EternalOrchestrator(context):
          while True:
              message = await context.wait_for_external_event('NewMessage')
              await context.call_activity('ProcessMessage', message)
              await context.call_activity('WriteToOutputQueue', message)
              if context.is_terminating():
                  context.continue_as_new(None)
                  break
   

   This method ensures that the orchestration completes its current tasks before resetting. Read more about it here.
2. You can use External Events to signal the orchestration to stop processing new messages. This involves setting up an external event that the orchestration listens for and then gracefully stops when the event is received.

      async def EternalOrchestrator(context):
          while True:
              message = await context.wait_for_external_event('NewMessage')
              await context.call_activity('ProcessMessage', message)
              await context.call_activity('WriteToOutputQueue', message)
              stop_signal = await context.wait_for_external_event('StopSignal')
              if stop_signal:
                  break
   

   This method allows you to control the orchestration externally One of the good answers on StackOverflow.
3. The Durable Functions Monitor can be used to manually terminate an orchestration. This tool can be used as a VS Code extension, standalone service, or injected mode in a .NET Functions project - https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-instance-management

      func durable terminate <instanceId> --reason "Graceful stop"
   

   This command allows you to terminate the orchestration manually.

I hope this is helpful! Do not hesitate to let me know if you have any other questions or clarifications.

---

Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful.