Shutdown hooks cannot be modified during shutdown

Question

What does the ADF pipeline error message mean?

'Shutdown hooks cannot be modified during shutdown'.

Answer 1

The error means that the system is in the process of shutting down (or has already initiated shutdown procedures), and a process within the ADF pipeline or its components is attempting to modify a shutdown hook. A shutdown hook is a process that is executed when the system is shutting down, often used to clean up resources or gracefully terminate processes.

In the context of ADF, this error could occur in the following scenarios:

1. If your pipeline was in progress and the system or runtime is being shut down (perhaps due to scheduled maintenance or an unexpected issue), and a resource (such as a file, data stream, or external system) tries to modify its shutdown hook, this error may occur.
2. If the pipeline is terminated unexpectedly (like canceled by a user, external failure), and an operation within the pipeline, such as data movement or transformation, attempts to set up a process for cleanup or retry, it may fail to do so because the shutdown process has already begun.
3. If there are system-level issues such as resource exhaustion or a timeout, the ADF process may have initiated a shutdown, leading to this error if an ongoing operation tries to register a shutdown hook after shutdown has been triggered.

What to do ?

• Look into the pipeline logs to identify if a shutdown was initiated by Azure (during a maintenance window) or if it was due to an unexpected termination.
• Ensure that your pipeline is not exhausting resources (memory, CPU) and causing a shutdown. Azure Monitoring tools can help you check for resource constraints.
• Implement retry logic in your pipeline to handle transient failures gracefully.
• If the pipeline ran longer than expected and hit a timeout, ensure that your pipeline activities are correctly configured to handle longer running processes.