Graceful shutdown of app service on Linux

Question

Graceful shutdown of app service on Linux

Ad Gaasbeek 0

I have a Azure App Service (Linux, Java).

When I scale down the number of instances, my application gets killed immediately. However, this leads to transaction errors and database locks. What I want to accomplish is to detect the shutdown, and being able to finish the ongoing work (while not starting any new processes and let my health-check return 503)/

I am able to detect receiving the TERM signal and starting my shutdown process. But Azure does not wait for this to complete, but apparently decides to immediately kill the application.

How can I make Azure to wait killing my application until I say my application is ready for it?

PS When stopping my application, it also does not respect the handling of the TERM signal, and kills immediately.

3 answers

Your answer

Answer 1

Laxman Reddy Revuri 5,405 Microsoft External Staff Moderator

Hi @Ad Gaasbeek
Thanks for the question and using MS Q&A platform.
Unfortunately, Azure App Service does not provide a built-in mechanism to extend the shutdown timeout beyond the default behavior, which may lead to immediate termination of your application. Therefore, you may need to handle the shutdown logic within your application to minimize the impact of abrupt terminations.

Additionally for smooth shut down your Azure App Service on Linux, make sure to handle the TERM signal properly. Keep in mind that Azure App Service doesn't automatically wait for your app to finish shutting down before it terminates.

Set up health probes to manage the app's state during the shutdown. By returning a 503-status code from your health checks when you get a shutdown signal, you can stop new requests while letting ongoing tasks wrap up. But note, Azure might still enforce a timeout period and forcibly terminate your app if it exceeds this period.

1.Implement signal handling to catch the TERM signal.

2.Use health probes to show when the app isn't ready for new traffic.

3.Ensure the app finishes ongoing transactions before the timeout hits.
references:
https://learn.microsoft.com/en-us/azure/spring-apps/basic-standard/how-to-configure-health-probes-graceful-termination?tabs=azure-portal#health-probe-properties
Please accept as "Yes" if the answer provided is useful, so that you can help others in the community looking for remediation for similar issues.

Laxman Reddy Revuri 5,405 Reputation points Microsoft External Staff Moderator

2025-02-03T16:04:12.6433333+00:00

Hi @Ad Gaasbeek
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Ad Gaasbeek 0 Reputation points

2025-02-04T06:58:50.93+00:00

Thank you very much for your answer.

A few things are still unclear to me though. If I do handle the TERM signal properly, is it then guaranteed that Azure waits till this handling is finished before it kills my App Service? (within the 30 seconds timeout). I tested it, and the answer seems no, but I might be doing something wrong. So before I spend more time one, I want to be absolutely sure that it is possible.

The link you provide mentions a health probe. First of all, I can not find this for a Linux App Service. And secondly, this serves another purpose, determining if my application is healthy. When stopping it myself or through auto-scaling, I think that the health probe will have no influence whatsoever.
Laxman Reddy Revuri 5,405 Reputation points Microsoft External Staff Moderator

2025-02-04T09:05:41.1866667+00:00

Hi @Ad Gaasbeek
apologize for any inconvenience caused on this.
Azure App Service doesn't guarantee waiting for your app to finish handling the TERM signal before shutting down. You can set up signal handling for a smooth shutdown, but Azure might still enforce a timeout (usually around 30 seconds) and forcefully terminate your app if it hasn't shut down by then. Health probes mainly check your app's health, but you can configure them to return a 503 status during shutdown to stop new requests. However, health probes might not be available for Linux App Services like they are for other services, limiting their effectiveness in managing shutdowns. To sum it up, you should: Implement signal handling for the TERM signal, use health probes to show when the app isn't ready for new traffic (if possible), and make sure your app can finish ongoing tasks within the timeout to avoid abrupt termination.
Ad Gaasbeek 0 Reputation points

2025-02-05T09:54:24.8066667+00:00
I tried it out. I am handing the TERM signal:

log when entering the handler method

sleep 5 seconds

log message

repeat 3 times

log when exiting the handler method

Result: Azure kills the App Service immediately, I only see the first log message about entering my handler. When running it locally on my laptop, every works fine and I see all the log messages.

How to set the time that Azure waits for the TERM signal handling?

What is the default? If it is 30 seconds, than it clearly does not work (as expected)
Laxman Reddy Revuri 5,405 Reputation points Microsoft External Staff Moderator

2025-02-05T19:10:14.0633333+00:00
Hi @Ad Gaasbeek
To ensure your application gracefully shuts down on Azure App Service (Linux)
IHostApplicationLifetime manages clean shut down of your client application.
Enables the application to accept shutdown events and to do some cleanup actions (for example finishing transactions, freeing resources).
Use ApplicationStopping event to register for a shutdown callback.
Do finishing activities like ongoing requests or transactions.

public class Startup{ public void Configure(IApplicationBuilder app, IHostApplicationLifetime applicationLifetime) { applicationLifetime.ApplicationStopping.Register(() => { // Cleanup logic (e.g., finish transactions) Console.WriteLine("Application is stopping..."); }); // Other middleware setup... } }

Handling the Signal Termination (SIGTERM) Azure sends a SIGTERM signal to the container whenever the application is scaling down or shutting down.
Your application needs to respond to this signal by performing necessary tasks required to shut it down properly, such as canceling some active requests or rolling back or removing DML database changes.
Azure App Service provides a default short grace period during which the application is expected to shut down gracefully (usually 30 seconds before force stopping the application).
Set the WEBSITES_SHUTDOWN_TIME environment variable if a full shutdown during the processed needed time exceeds the default, offering more time to the app for completion of essential actions.

WEBSITES_SHUTDOWN_TIME=60

The shutdown timeout to 60 seconds, giving your app extended time to complete promises being executed. Application Ordering and Response Request Handling Terminates all new requests during the shutdown process.
During the shutdown process, ensure that your application:

Stops accepting new requests.

Returns 503 Service Unavailable during the shutdown period using health checks.

This prevents Azure from sending requests to your app while it is in a shutdown state.
https://learn.microsoft.com/en-us/aspnet/core/fundamentals/host/web-host?view=aspnetcore-9.0
Ad Gaasbeek 0 Reputation points

2025-02-06T07:43:46.14+00:00

DELETED

Ad Gaasbeek 0

The example you give with IHostApplicationLifetime is C# not Java. Since I use Quarkus, I cannot use this. I use the code below for testing:

@Startup
@Singleton
@Slf4j
public class ShutdownHandler {
    @PostConstruct
    public void init() {
        log.info("TERM: setting up shutdown handler");
        Signal.handle(new Signal("TERM"), sig -> handleShutdown());
    }

    public void handleShutdown() {
        log.info("TERM: Shutting down");

        for (int i = 0; i < 3; i++) {
            log.info("TERM: Shutting down {}", i);
            try {
                // stop processes, for now just use sleep
                Thread.sleep(5000);
            } catch (InterruptedException e) {
                log.error("TERM: Error while sleeping", e);
            }
        }

        log.info("TERM: exit(0)");
        System.exit(0);
    }

I see the setting up logging and the first two shutting down log statements. After the first sleep call, the application is immediately killed.

What should I use to wait for my processes to finish? It would be great if you have some example code that actually works.

The setting WEBSITES_SHUTDOWN_TIME does not help. The default of 30 seconds would be sufficient for me anyway. Where can I found documentation about this setting?

Laxman Reddy Revuri 5,405 Reputation points Microsoft External Staff Moderator

2025-02-06T18:38:05.03+00:00

Hi @Ad Gaasbeek
apologize for any inconvenience caused on this.
I understand that how to properly shut down your Quarkus app that is running on Azure App Service (Linux). You have a SIGTERM signal handler, but the app is shutting off before it fully shuts down.

The application is free to stop taking on new requests while existing ones are still being processed, a capability which Quarkus natively supports. This is controlled by the quarkus.shutdown.timeout setting.

append the application’s properties file with the following keywords without This instructs Quarkus to allow up to 30 seconds for active requests to complete during a shutdown.
quarkus.shutdown.timeout=30s

Quarkus automatically handles SIGTERM signals and commences the shutdown procedure. Be sure your app components, say transactions or certain backend jobs, finish properly within the allotted period.
https://quarkus.io/guides/lifecycle#the-main-method

The maximum time the Azure App Service waits for your application to shut down after receiving the request is specified in this environment variable. Default is 30 seconds.

Set Environment Variable:

If your app requires more time to stop, then raise this timeout in the Azure App Service settings.

References: Extend timeout for graceful shutdown in Azure App Service - Microsoft Q&A Sync Timeouts

Ensure quarkus.shutdown.timeout set enough to give your application time to shut down. For example, if your app requires 60 seconds, increase both to 60 seconds.

Use health probes to manage the app's state during shutdown. By returning a 503-status code from your health checks when you get a shutdown signal, you can stop new requests while finishing ongoing tasks.
Laxman Reddy Revuri 5,405 Reputation points Microsoft External Staff Moderator

2025-02-07T17:14:06.6333333+00:00

Hi @Ad Gaasbeek
Following up to see if you have chance to check my previous response and help us with requested information to check and assist you further on this.
Laxman Reddy Revuri 5,405 Reputation points Microsoft External Staff Moderator

2025-02-10T09:41:57.1333333+00:00

Hi @Ad Gaasbeek
Following up to see if you have chance to check my previous response and help us with requested information to check and assist you further on this.

Answer 2

Ad Gaasbeek 0

It seems that I can solve my problem by:

setting the environment variable WEBSITES_SHUTDOWN_TIME (can not find it in the documentation)
in my @Startup code handle the term signal:
Signal.handle(new Signal("TERM"), sig -> handleShutdown());
in the application properties set quarkus.shutdown.timeout

Laxman Reddy Revuri 5,405 Reputation points Microsoft External Staff Moderator

2025-02-18T08:55:58.5566667+00:00

Hi @Ad Gaasbeek
Could you please let me know if your issue has been resolved, or if you still need any assistance?
Ad Gaasbeek 0 Reputation points

2025-02-18T09:05:57.74+00:00

Unfortunately, as it turns out, it has not been resolved yet.
My solution works when I hit the stop button in the UI. Then Azure respects the configured timeout. But, when the application is down-scaled, then Azure still kills the application immediately.
So, the question remains, how to stop Azure from killing my app service immediately and ignoring the timeout setting.
Harshitha Veeramalla 1,306 Reputation points Microsoft External Staff Moderator

2025-02-18T14:44:40.74+00:00

Hi @Ad Gaasbeek,
Is your Always on option is enabled in Configuration section?
Ad Gaasbeek 0 Reputation points

2025-02-18T15:05:53.93+00:00

No, it is disabled (Off)
Harshitha Veeramalla 1,306 Reputation points Microsoft External Staff Moderator

2025-02-18T15:07:36.2433333+00:00

Can you please enable it and check the behavior of the app once ?
Harshitha Veeramalla 1,306 Reputation points Microsoft External Staff Moderator

2025-02-20T06:38:59.7266667+00:00

Ad Gaasbeek Any update on the issue?
Ad Gaasbeek 0 Reputation points

2025-05-26T11:52:57.1133333+00:00

On or off doesn't make a difference.

Answer 3

I asked Senior Cloud Solution Architect from Microsoft, and this was his answer:

I looked into the methods on how Linux on App Service operates, but generally speaking, as you already attest, it lacks the following:

Azure App Service for Linux does not guarantee a graceful shutdown during scale-in operations. Specifically:

Immediate Termination: During scale-in, Azure may send a SIGTERM, but often follows up with a SIGKILL almost immediately, leaving little to no time for the application to perform cleanup operations.
WEBSITES_SHUTDOWN_TIMEOUT Ineffectiveness: Setting the WEBSITES_SHUTDOWN_TIMEOUT environment variable does not reliably extend the shutdown period during scale-in events.
Health Probes Limitations: While health probes can indicate application readiness, they do not influence the shutdown behavior during scale-in on Linux App Services.

Therefore, I would suggest for Linux to look into alternative, and from my point of view much better, solutions, like Container Apps, which gives you an actual control plane for that, which you control.

Share via

Graceful shutdown of app service on Linux

3 answers

Your answer