Share via

Azure Pipelines Cancel Leaves Orphan Processes

Alexandru Gheara 5 Reputation points
2024-06-20T10:16:03.9866667+00:00

I am reaching out to seek assistance with an issue we are encountering with our Azure DevOps pipelines. We are experiencing a situation where our pipelines, when canceled, do not properly terminate running processes, leading to orphaned processes that continue executing. Despite the job being marked as canceled in the Azure DevOps interface, the elapsed time continues, and the script logs keep generating.

Background

Here is a brief overview of our setup:

  • We are running a Python script that utilizes ThreadPoolExecutor for parallel execution.
  • The script includes signal handling to catch SIGINT and SIGTERM signals to ensure graceful shutdowns.
  • The environment is Kubernetes-based, with Azure DevOps agents running in an EKS cluster.

Problem Description

When a pipeline is canceled from the Azure DevOps UI:

  1. The job is marked as canceled, but the running script continues executing.
  2. The elapsed time for the job continues to increase.
  3. Logs from the running script continue to be generated, indicating that the script is still running.

We have configured our script to handle termination signals and tested it in local environments where it behaves as expected. However, in the Azure DevOps environment, it seems the termination signals are not being propagated to the script properly.

Request for Assistance

We would greatly appreciate your assistance in resolving this issue. Specifically, we need guidance on:

  1. Ensuring that Azure DevOps correctly propagates termination signals to running scripts.
  2. Any additional configuration or best practices to prevent orphaned processes when pipelines are canceled.

Thank you for your support and looking forward to your response.

Best regards,

[Alexandru Gheara]

Not Monitored
Not Monitored
Tag not monitored by Microsoft.
37,674 questions
0 comments

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. PRADEEPCHEEKATLA-MSFT 85,351 Reputation points Microsoft Employee
    2024-06-25T10:30:31.55+00:00

    @Alexandru Gheara - Thanks for the question and using MS Q&A platform.

    Azure DevOps is currently not supported in the Microsoft Q&A platform; the supported products are listed over here https://docs.microsoft.com/en-us/answers/products (more to be added later on).

    In order to assist best on your query, I would request you to post your query in SO => Azure Devops dedicated support. Additionally, adding the [Azure] tag on SO will increase visibility as it is a Microsoft Sponsored tag.

    https://stackoverflow.com/questions/tagged/azure-devops

    OR

    Report any Azure Devops problems on Developer Community.

    This will assist you with a faster reply to your query.

    0 comments