Automating Docker Image Patching for Azure DevOps Agent Pools

Question

Problem Statement: Automate the patching of Docker images for agent pools in Azure DevOps using a pipeline.

Proposed Solution:

Develop a Python script that utilizes REST API calls to manage Azure DevOps agent pools through the following steps:

1. Check if the agents are running on the latest image tag.
2. Identify idle agents.

• Create batches of idle agents (10% of total agents).

1. Identify busy agents.

• Disable busy agents.
  • Wait for them to become idle.
    • Proceed once they are idle.
    1. Identify offline agents.
    • No action required for offline agents.
    1. For the steps involving busy and offline agents, implement sequential thread processing in a loop until all batches have been processed:
    • For each batch:
      • Drain agents.
        • Check after 1 minute to ensure they are drained/removed from the pool.
          • Mark processed agents and log this in file1 (YAML/JSON).
            • Trigger a pipeline to reinstate agents and check their status after 30 minutes.
              • Verify if the agents are operational.
                • Log the newly added agents in file2 (YAML/JSON).
                  • Proceed to the next batch.
                  1. Implement retry logic, rollback changes, and log details of the agents.
                  2. Ensure both file1 and file2 match the agent names.

Seeking suggestions for possible solutions and tests that could be included to achieve this automation.