DevOps build pipelines waiting for Microsoft hosted agent

Question

This morning I discovered that we had 48 build pipelines that had been waiting for a hosted agent for more than five hours.

The jobs were stuck with the following message:

"This agent request is not running yet. Current position in queue: 1. Current parallelism: 0. Max Parallelism for parallelism type 'Microsoft-Hosted Private': 20"

I have tried cancelling all the jobs, and started a new, but it doesn't get an agent assigned.

I have also tried disabling all of the 20 agents, and enabling them again. But it doesn't make any difference.

We have no jobs that have been running since midnight, so it seems like an issue with the Microsoft hosted agents.

Answer 1

From a cursory check, I cant see any notification of an issue with Azure: https://status.dev.azure.com/

This issue usually means MS agents are occupied.

Can you confirm we havent exceeded the allocated parallel jobs?

• You can check your parallel job limits in Azure DevOps Organization Settings > Pipelines > Parallel jobs.
• Make sure that the quota for parallel jobs has not been exceeded and that no other pipelines are occupying the available slots. See: https://learn.microsoft.com/en-us/answers/questions/2201281/all-of-my-pipelines-stopped-working-on-friday-stuc

You can further try the steps in the link above.

If they do not work, I would recommend you contact support for this.

Also see: https://learn.microsoft.com/en-us/azure/devops/pipelines/troubleshooting/troubleshoot-start?view=azure-devops&source=recommendations

You can mark it 'Accept Answer' and 'Upvote' if this helped you

Regards,

Abiola

Answer 2

Morten Jensen, I completely understand how frustrating it can be to see dozens of build pipelines stuck in queue for hours on end, especially when everything looks online and there does not appear to be any obvious reason. What happened here is not a mistake on your part. In fact, Azure’s shared pool of Microsoft-hosted VMs in your region was fully booked by other teams across DevOps, which meant none of your queued pipelines could start until more capacity became available. Once Azure added additional VMs around 9:10 AM, all your pipelines immediately began running.

When you navigate to Organization Settings and look under Pipelines › Parallel jobs › Microsoft-hosted, you might see that your “In use” count falls below the “Total” slots allocated to your account. That can be misleading, because Azure DevOps does not instantly provision a new VM whenever a slot is technically available. During peak demand, every VM in that pool may already be in use by other customers, so even if your project is not actively consuming all its slots, your build will remain queued until Microsoft brings additional VMs online.

To free up a slot quickly when you see a stuck run holding onto a hosted agent, you can cancel it using the Azure DevOps CLI. For example, if your organization URL is https://dev.azure.com/Contoso and your project name is WebApp.

az pipelines runs cancel \
  --id 12345 \
  --org https://dev.azure.com/Contoso \
  --project WebApp

Replace 12345 with the actual run ID you find under Pipelines › Runs. Once that slot is released, your next queued build will pick it up automatically.

However, the most reliable way to guarantee build capacity particularly for mission-critical pipelines is to set up your own self-hosted agent. When you run an agent on a virtual machine you control (for instance, a Linux VM running Ubuntu 20.04), it never competes with the shared pool, so your pipeline can start immediately whenever you need it. Here is a quick walkthrough to get a Linux-based self-hosted agent up and running:

• Choose or create a Linux VM (Ubuntu 20.04 or later is ideal) and SSH into it.
• Create a directory for the agent, download the agent package, extract it, and configure it:

mkdir ~/azure-agent && cd ~/azure-agent
wget https://vstsagentpackage.azureedge.net/agent/2.255.1/vsts-agent-linux-x64-2.255.1.tar.gz
tar zxvf vsts-agent-linux-x64-2.255.1.tar.gz
./config.sh \
  --unattended \
  --url https://dev.azure.com/Contoso \
  --auth pat \
  --token YOUR_PERSONAL_ACCESS_TOKEN \
  --pool MySelfHostedPool \
  --agent MyAgentName

In this command, replace https://dev.azure.com/Contoso with your actual organization URL and supply a Personal Access Token (PAT) that has permission to register agents. You can name the pool and agent whatever makes sense just remember those names for your pipeline definition.

• Next, turn this agent into a background service so it stays online across reboots. Create a systemd unit file with the following content (adjusting paths as needed)

sudo tee /etc/systemd/system/azuredevops-agent.service <<EOF
[Unit]
Description=Azure Pipelines Agent
After=network.target
[Service]
User=$(whoami)
WorkingDirectory=/home/$(whoami)/azure-agent
ExecStart=/home/$(whoami)/azure-agent/run.sh
Restart=always
RestartSec=3
[Install]
WantedBy=default.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable azuredevops-agent
sudo systemctl start azuredevops-agent

After this, you should see your new agent appear as “Online” under Organization Settings › Pipelines › Agent pools in Azure DevOps. As soon as it is online, point any critical YAML pipeline at this private pool by specifying pool: MySelfHostedPool (or select that pool in the Classic editor). From that moment on, your builds will never have to wait on the shared Microsoft-hosted queue.

If you are working on Windows instead, the steps are similar: download the Windows agent ZIP, extract it, run config.cmd --unattended --url https://dev.azure.com/Contoso --auth pat --token YOUR_PERSONAL_ACCESS_TOKEN --pool MySelfHostedPool, then install and start it as a Windows service by running .\svc.sh install and .\svc.sh start. Once your agent goes online, you can assign pipelines to it in exactly the same way.

If you must continue using Microsoft-hosted agents for all pipelines, you can reduce the likelihood of a long queue by experimenting with different VM images for example, switch from windows-2022 to ubuntu-22.04 because one pool may temporarily have more free nodes than another. You can also consider purchasing additional parallel-job slots under Organization Settings › Billing › Parallel jobs. Buying extra slots raises your account’s priority when Azure scales out new VMs, although it does not completely eliminate the chance of a bottleneck under peak demand.

If you ever see hosted jobs stuck for more than a few hours again, open a support ticket with Azure DevOps. Include the timestamps when jobs entered the queue, the pipeline or run IDs, and the specific VM images you requested. This helps Microsoft confirm any regional capacity constraints and add more nodes for your tenants more quickly.