I am reaching out to seek assistance with an issue we are encountering with our Azure DevOps pipelines. We are experiencing a situation where our pipelines, when canceled, do not properly terminate running processes, leading to orphaned processes that continue executing. Despite the job being marked as canceled in the Azure DevOps interface, the elapsed time continues, and the script logs keep generating.
Background
Here is a brief overview of our setup:
- We are running a Python script that utilizes
ThreadPoolExecutor
for parallel execution.
- The script includes signal handling to catch SIGINT and SIGTERM signals to ensure graceful shutdowns.
- The environment is Kubernetes-based, with Azure DevOps agents running in an EKS cluster.
Problem Description
When a pipeline is canceled from the Azure DevOps UI:
- The job is marked as canceled, but the running script continues executing.
- The elapsed time for the job continues to increase.
- Logs from the running script continue to be generated, indicating that the script is still running.
We have configured our script to handle termination signals and tested it in local environments where it behaves as expected. However, in the Azure DevOps environment, it seems the termination signals are not being propagated to the script properly.
Request for Assistance
We would greatly appreciate your assistance in resolving this issue. Specifically, we need guidance on:
- Ensuring that Azure DevOps correctly propagates termination signals to running scripts.
- Any additional configuration or best practices to prevent orphaned processes when pipelines are canceled.
Thank you for your support and looking forward to your response.
Best regards,
[Alexandru Gheara]