Run a self-hosted agent in Docker

Article
04/05/2024

Azure DevOps Services | Azure DevOps Server 2022 - Azure DevOps Server 2019

This article provides instructions for running your Azure Pipelines agent in Docker. You can set up a self-hosted agent in Azure Pipelines to run inside a Windows Server Core (for Windows hosts), or Ubuntu container (for Linux hosts) with Docker. This is useful when you want to run agents with outer orchestration, such as Azure Container Instances. In this article, you'll walk through a complete container example, including handling agent self-update.

Both Windows and Linux are supported as container hosts. Windows containers should run on a Windows vmImage. To run your agent in Docker, you'll pass a few environment variables to docker run, which configures the agent to connect to Azure Pipelines or Azure DevOps Server. Finally, you customize the container to suit your needs. Tasks and scripts might depend on specific tools being available on the container's PATH, and it's your responsibility to ensure that these tools are available.

This feature requires agent version 2.149 or later. Azure DevOps 2019 didn't ship with a compatible agent version. However, you can upload the correct agent package to your application tier if you want to run Docker agents.

Windows

Enable Hyper-V

Hyper-V isn't enabled by default on Windows. If you want to provide isolation between containers, you must enable Hyper-V. Otherwise, Docker for Windows won't start.

Note

You must enable virtualization on your machine. It's typically enabled by default. However, if Hyper-V installation fails, refer to your system documentation for how to enable virtualization.

Install Docker for Windows

If you're using Windows 10, you can install the Docker Community Edition. For Windows Server 2016, install the Docker Enterprise Edition.

Switch Docker to use Windows containers

By default, Docker for Windows is configured to use Linux containers. To allow running the Windows container, confirm that Docker for Windows is running the Windows daemon.

Create and build the Dockerfile

Next, create the Dockerfile.

Open a command prompt.
Create a new directory:
```
mkdir "C:\azp-agent-in-docker\"
```
Go to this new directory:
```
cd "C:\azp-agent-in-docker\"
```

Save the following content to a file called C:\azp-agent-in-docker\azp-agent-windows.dockerfile:

FROM mcr.microsoft.com/windows/servercore:ltsc2022

WORKDIR /azp/

COPY ./start.ps1 ./

CMD powershell .\start.ps1

Save the following content to C:\azp-agent-in-docker\start.ps1:

function Print-Header ($header) {
  Write-Host "`n${header}`n" -ForegroundColor Cyan
}

if (-not (Test-Path Env:AZP_URL)) {
  Write-Error "error: missing AZP_URL environment variable"
  exit 1
}

if (-not (Test-Path Env:AZP_TOKEN_FILE)) {
  if (-not (Test-Path Env:AZP_TOKEN)) {
    Write-Error "error: missing AZP_TOKEN environment variable"
    exit 1
  }

  $Env:AZP_TOKEN_FILE = "\azp\.token"
  $Env:AZP_TOKEN | Out-File -FilePath $Env:AZP_TOKEN_FILE
}

Remove-Item Env:AZP_TOKEN

if ((Test-Path Env:AZP_WORK) -and -not (Test-Path $Env:AZP_WORK)) {
  New-Item $Env:AZP_WORK -ItemType directory | Out-Null
}

New-Item "\azp\agent" -ItemType directory | Out-Null

# Let the agent ignore the token env variables
$Env:VSO_AGENT_IGNORE = "AZP_TOKEN,AZP_TOKEN_FILE"

Set-Location agent

Print-Header "1. Determining matching Azure Pipelines agent..."

$base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(":$(Get-Content ${Env:AZP_TOKEN_FILE})"))
$package = Invoke-RestMethod -Headers @{Authorization=("Basic $base64AuthInfo")} "$(${Env:AZP_URL})/_apis/distributedtask/packages/agent?platform=win-x64&`$top=1"
$packageUrl = $package[0].Value.downloadUrl

Write-Host $packageUrl

Print-Header "2. Downloading and installing Azure Pipelines agent..."

$wc = New-Object System.Net.WebClient
$wc.DownloadFile($packageUrl, "$(Get-Location)\agent.zip")

Expand-Archive -Path "agent.zip" -DestinationPath "\azp\agent"

try {
  Print-Header "3. Configuring Azure Pipelines agent..."

  .\config.cmd --unattended `
    --agent "$(if (Test-Path Env:AZP_AGENT_NAME) { ${Env:AZP_AGENT_NAME} } else { hostname })" `
    --url "$(${Env:AZP_URL})" `
    --auth PAT `
    --token "$(Get-Content ${Env:AZP_TOKEN_FILE})" `
    --pool "$(if (Test-Path Env:AZP_POOL) { ${Env:AZP_POOL} } else { 'Default' })" `
    --work "$(if (Test-Path Env:AZP_WORK) { ${Env:AZP_WORK} } else { '_work' })" `
    --replace

  Print-Header "4. Running Azure Pipelines agent..."

  .\run.cmd
} finally {
  Print-Header "Cleanup. Removing Azure Pipelines agent..."

  .\config.cmd remove --unattended `
    --auth PAT `
    --token "$(Get-Content ${Env:AZP_TOKEN_FILE})"
}

Run the following command within that directory:
```
docker build --tag "azp-agent:windows" --file "./azp-agent-windows.dockerfile" .
```
The final image is tagged azp-agent:windows.

Start the image

Now that you have created an image, you can run a container. This installs the latest version of the agent, configures it, and runs the agent. It targets the specified agent pool (the Default agent pool by default) of a specified Azure DevOps or Azure DevOps Server instance of your choice:

docker run -e AZP_URL="<Azure DevOps instance>" -e AZP_TOKEN="<Personal Access Token>" -e AZP_POOL="<Agent Pool Name>" -e AZP_AGENT_NAME="Docker Agent - Windows" --name "azp-agent-windows" azp-agent:windows

You might need to specify the --network parameter if you run into network issues.

docker run --network "Default Switch" < . . . >

You might need to specify --interactive and --tty flags (or simply -it) if you want to be able to stop the container and remove the agent with Ctrl + C.

docker run --interactive --tty < . . . >

If you want a fresh agent container for every pipeline job, pass the --once flag to the run command.

docker run < . . . > --once

With the --once flag, you might want to use a container orchestration system, like Kubernetes or Azure Container Instances, to start a new copy of the container when the job completes.

You can control the agent name, the agent pool, and the agent work directory by using optional environment variables.

Linux

Install Docker

Depending on your Linux Distribution, you can either install Docker Community Edition or Docker Enterprise Edition.

Create and build the Dockerfile

Next, create the Dockerfile.

Open a terminal.
Create a new directory (recommended):
```
mkdir ~/azp-agent-in-docker/
```
Go to this new directory:
```
cd ~/azp-agent-in-docker/
```

Save the following content to ~/azp-agent-in-docker/azp-agent-linux.dockerfile:

For Alpine:

FROM alpine

RUN apk update
RUN apk upgrade
RUN apk add bash curl git icu-libs jq

ENV TARGETARCH="linux-musl-x64"

WORKDIR /azp/

COPY ./start.sh ./
RUN chmod +x ./start.sh

RUN adduser -D agent
RUN chown agent ./
USER agent
# Another option is to run the agent as root.
# ENV AGENT_ALLOW_RUNASROOT="true"

ENTRYPOINT ./start.sh

For Ubuntu 22.04:

FROM ubuntu:22.04

RUN apt update -y && apt upgrade -y && apt install curl git jq libicu70 -y

# Also can be "linux-arm", "linux-arm64".
ENV TARGETARCH="linux-x64"

WORKDIR /azp/

COPY ./start.sh ./
RUN chmod +x ./start.sh

# Create agent user and set up home directory
RUN useradd -m -d /home/agent agent
RUN chown -R agent:agent /azp /home/agent

USER agent
# Another option is to run the agent as root.
# ENV AGENT_ALLOW_RUNASROOT="true"

ENTRYPOINT ./start.sh

Uncomment the ENV AGENT_ALLOW_RUNASROOT="true" line and remove adding the agent user before this line if you want to run the agent as root.

Note

Tasks might depend on executables that your container is expected to provide. For instance, you must add the zip and unzip packages to the RUN apt install -y command in order to run the ArchiveFiles and ExtractFiles tasks. Also, as this is a Linux Ubuntu image for the agent to use, you can customize the image as you need. E.g.: if you need to build .NET applications you can follow the document Install the .NET SDK or the .NET Runtime on Ubuntu and add that to your image.

Save the following content to ~/azp-agent-in-docker/start.sh, making sure to use Unix-style (LF) line endings:

#!/bin/bash
set -e

if [ -z "${AZP_URL}" ]; then
  echo 1>&2 "error: missing AZP_URL environment variable"
  exit 1
fi

if [ -z "${AZP_TOKEN_FILE}" ]; then
  if [ -z "${AZP_TOKEN}" ]; then
    echo 1>&2 "error: missing AZP_TOKEN environment variable"
    exit 1
  fi

  AZP_TOKEN_FILE="/azp/.token"
  echo -n "${AZP_TOKEN}" > "${AZP_TOKEN_FILE}"
fi

unset AZP_TOKEN

if [ -n "${AZP_WORK}" ]; then
  mkdir -p "${AZP_WORK}"
fi

cleanup() {
  trap "" EXIT

  if [ -e ./config.sh ]; then
    print_header "Cleanup. Removing Azure Pipelines agent..."

    # If the agent has some running jobs, the configuration removal process will fail.
    # So, give it some time to finish the job.
    while true; do
      ./config.sh remove --unattended --auth "PAT" --token $(cat "${AZP_TOKEN_FILE}") && break

      echo "Retrying in 30 seconds..."
      sleep 30
    done
  fi
}

print_header() {
  lightcyan="\033[1;36m"
  nocolor="\033[0m"
  echo -e "\n${lightcyan}$1${nocolor}\n"
}

# Let the agent ignore the token env variables
export VSO_AGENT_IGNORE="AZP_TOKEN,AZP_TOKEN_FILE"

print_header "1. Determining matching Azure Pipelines agent..."

AZP_AGENT_PACKAGES=$(curl -LsS \
    -u user:$(cat "${AZP_TOKEN_FILE}") \
    -H "Accept:application/json;" \
    "${AZP_URL}/_apis/distributedtask/packages/agent?platform=${TARGETARCH}&top=1")

AZP_AGENT_PACKAGE_LATEST_URL=$(echo "${AZP_AGENT_PACKAGES}" | jq -r ".value[0].downloadUrl")

if [ -z "${AZP_AGENT_PACKAGE_LATEST_URL}" -o "${AZP_AGENT_PACKAGE_LATEST_URL}" == "null" ]; then
  echo 1>&2 "error: could not determine a matching Azure Pipelines agent"
  echo 1>&2 "check that account "${AZP_URL}" is correct and the token is valid for that account"
  exit 1
fi

print_header "2. Downloading and extracting Azure Pipelines agent..."

curl -LsS "${AZP_AGENT_PACKAGE_LATEST_URL}" | tar -xz & wait $!

source ./env.sh

trap "cleanup; exit 0" EXIT
trap "cleanup; exit 130" INT
trap "cleanup; exit 143" TERM

print_header "3. Configuring Azure Pipelines agent..."

./config.sh --unattended \
  --agent "${AZP_AGENT_NAME:-$(hostname)}" \
  --url "${AZP_URL}" \
  --auth "PAT" \
  --token $(cat "${AZP_TOKEN_FILE}") \
  --pool "${AZP_POOL:-Default}" \
  --work "${AZP_WORK:-_work}" \
  --replace \
  --acceptTeeEula & wait $!

print_header "4. Running Azure Pipelines agent..."

chmod +x ./run.sh

# To be aware of TERM and INT signals call ./run.sh
# Running it with the --once flag at the end will shut down the agent after the build is executed
./run.sh "$@" & wait $!

Note

You must also use a container orchestration system, like Kubernetes or Azure Container Instances, to start new copies of the container when the work completes.

Run the following command within that directory:
```
docker build --tag "azp-agent:linux" --file "./azp-agent-linux.dockerfile" .
```
The final image is tagged azp-agent:linux.

Start the image

docker run -e AZP_URL="<Azure DevOps instance>" -e AZP_TOKEN="<Personal Access Token>" -e AZP_POOL="<Agent Pool Name>" -e AZP_AGENT_NAME="Docker Agent - Linux" --name "azp-agent-linux" azp-agent:linux

You might need to specify --interactive and --tty flags (or simply -it) if you want to be able to stop the container and remove the agent with Ctrl + C.

docker run --interactive --tty < . . . >

If you want a fresh agent container for every pipeline job, pass the --once flag to the run command.

docker run < . . . > --once

With the --once flag, you might want to use a container orchestration system, like Kubernetes or Azure Container Instances, to start a new copy of the container when the job completes.

You can control the agent name, the agent pool, and the agent work directory by using optional environment variables.

Environment variables

Environment variable	Description
AZP_URL	The URL of the Azure DevOps or Azure DevOps Server instance.
AZP_TOKEN	Personal Access Token (PAT) with Agent Pools (read, manage) scope, created by a user who has permission to configure agents, at `AZP_URL`.
AZP_AGENT_NAME	Agent name (default value: the container hostname).
AZP_POOL	Agent pool name (default value: `Default`).
AZP_WORK	Work directory (default value: `_work`).

Add tools and customize the container

You have created a basic build agent. You can extend the Dockerfile to include additional tools and their dependencies, or build your own container by using this one as a base layer. Just make sure that the following are left untouched:

The start.sh script is called by the Dockerfile.
The start.sh script is the last command in the Dockerfile.
Ensure that derivative containers don't remove any of the dependencies stated by the Dockerfile.

Use Docker within a Docker container

In order to use Docker from within a Docker container, you bind-mount the Docker socket.

Caution

Doing this has serious security implications. The code inside the container can now run as root on your Docker host.

If you're sure you want to do this, see the bind mount documentation on Docker.com.

Use Azure Kubernetes Service cluster

Caution

Please, consider that any docker based tasks will not work on AKS 1.19 or later due to docker in docker restriction. Docker was replaced with containerd in Kubernetes 1.19, and Docker-in-Docker became unavailable.

Deploy and configure Azure Kubernetes Service

Follow the steps in Quickstart: Deploy an Azure Kubernetes Service (AKS) cluster by using the Azure portal. After this, your PowerShell or Shell console can use the kubectl command line.

Deploy and configure Azure Container Registry

Follow the steps in Quickstart: Create an Azure container registry by using the Azure portal. After this, you can push and pull containers from Azure Container Registry.

Configure secrets and deploy a replica set

Create the secrets on the AKS cluster.

kubectl create secret generic azdevops \
  --from-literal=AZP_URL=https://dev.azure.com/yourOrg \
  --from-literal=AZP_TOKEN=YourPAT \
  --from-literal=AZP_POOL=NameOfYourPool

Run this command to push your container to Container Registry:
```
docker push "<acr-server>/azp-agent:<tag>"
```
Configure Container Registry integration for existing AKS clusters.
Note

If you have multiple subscriptions on the Azure Portal, please, use this command first to select a subscription
```
az account set --subscription "<subscription id or subscription name>"
```
```
az aks update -n "<myAKSCluster>" -g "<myResourceGroup>" --attach-acr "<acr-name>"
```

Save the following content to ~/AKS/ReplicationController.yml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: azdevops-deployment
  labels:
    app: azdevops-agent
spec:
  replicas: 1 # here is the configuration for the actual agent always running
  selector:
    matchLabels:
      app: azdevops-agent
  template:
    metadata:
      labels:
        app: azdevops-agent
    spec:
      containers:
      - name: kubepodcreation
        image: <acr-server>/azp-agent:<tag>
        env:
          - name: AZP_URL
            valueFrom:
              secretKeyRef:
                name: azdevops
                key: AZP_URL
          - name: AZP_TOKEN
            valueFrom:
              secretKeyRef:
                name: azdevops
                key: AZP_TOKEN
          - name: AZP_POOL
            valueFrom:
              secretKeyRef:
                name: azdevops
                key: AZP_POOL
        volumeMounts:
        - mountPath: /var/run/docker.sock
          name: docker-volume
      volumes:
      - name: docker-volume
        hostPath:
          path: /var/run/docker.sock

This Kubernetes YAML creates a replica set and a deployment, where replicas: 1 indicates the number or the agents that are running on the cluster.

Run this command:

kubectl apply -f ReplicationController.yml

Now your agents will run the AKS cluster.

Set custom MTU parameter

Allow specifying MTU value for networks used by container jobs (useful for docker-in-docker scenarios in k8s cluster).

You need to set the environment variable AGENT_DOCKER_MTU_VALUE to set the MTU value, and then restart the self-hosted agent. You can find more about agent restart here and about setting different environment variables for each individual agent here.

This allows you to set up a network parameter for the job container, the use of this command is similar to the use of the next command while container network configuration:

-o com.docker.network.driver.mtu=AGENT_DOCKER_MTU_VALUE

Mounting volumes using Docker within a Docker container

If a Docker container runs inside another Docker container, they both use host's daemon, so all mount paths reference the host, not the container.

For example, if we want to mount path from host into outer Docker container, we can use this command:

docker run ... -v "<path-on-host>:<path-on-outer-container>" ...

And if we want to mount path from host into inner Docker container, we can use this command:

docker run ... -v "<path-on-host>:<path-on-inner-container>" ...

But we can't mount paths from outer container into the inner one; to work around that, we have to declare an ENV variable:

docker run ... --env DIND_USER_HOME=$HOME ...

After this, we can start the inner container from the outer one using this command:

docker run ... -v "${DIND_USER_HOME}:<path-on-inner-container>" ...

Common errors

If you're using Windows, and you get the following error:

standard_init_linux.go:178: exec user process caused "no such file or directory"

Install Git Bash by downloading and installing git-scm.

Run this command:

dos2unix ~/azp-agent-in-docker/Dockerfile
dos2unix ~/azp-agent-in-docker/start.sh
git add .
git commit -m "Fixed CR"
git push

Try again. You no longer get the error.

Run a self-hosted agent in Docker

Windows

Enable Hyper-V

Install Docker for Windows

Switch Docker to use Windows containers

Create and build the Dockerfile

Start the image

Linux

Install Docker

Create and build the Dockerfile

Start the image

Environment variables

Add tools and customize the container

Use Docker within a Docker container

Use Azure Kubernetes Service cluster

Deploy and configure Azure Kubernetes Service

Deploy and configure Azure Container Registry

Configure secrets and deploy a replica set

Set custom MTU parameter

Mounting volumes using Docker within a Docker container

Common errors

Feedback

Feedback

Additional resources

Run a self-hosted agent in Docker

Windows

Enable Hyper-V

Install Docker for Windows

Switch Docker to use Windows containers

Create and build the Dockerfile

Start the image

Linux

Install Docker

Create and build the Dockerfile

Start the image

Environment variables

Add tools and customize the container

Use Docker within a Docker container

Use Azure Kubernetes Service cluster

Deploy and configure Azure Kubernetes Service

Deploy and configure Azure Container Registry

Configure secrets and deploy a replica set

Set custom MTU parameter

Mounting volumes using Docker within a Docker container

Common errors

Related articles

Feedback

Feedback

Additional resources