FDeploy Image is failing - AKS

Question

FDeploy Image is failing - AKS

Diptesh Kumar 666

Working scenario:

i have prepared the image using following commands in docker desktop which you can see below and pushed to ACR and then successfully deployed that image using AKS


USER@MACHINE:~$ docker tag aksimage:latest acrrrr.azurecr.io/aksimage:latest

USER@MACHINE:~$ docker push acrrrr.azurecr.io/aksimage:latest

The push refers to repository [acrrrr.azurecr.io/aksimage]

1fe6f44321a3: Pushed

936898278d02: Pushed

a702764f6c9e: Pushed

2dee78b1264b: Pushed

06e3bc688725: Pushed

2a58e20c885d: Pushed

86becefdf252: Pushed

65c636ce09f2: Pushed

latest: digest: sha256:466e1fe0b6b9316a5c8c72c37e72934de9cb2ccf3f717841e448d59cbd06cc9f size: 1992

FYI: I used ubhuntu 22.4 LTS since my laptops windows 11 in my case

Below is not working scenario :

where as here I am using same commands but I am running from pipeline and used same commands you can see below and I am using ubhuntu latest as agent which you can see below, here also image gets pushed but when I deploy I am seeing following error when deploy it using AKS


variables:

  imageName: 'aksimage'

  dockerfilePath: 'Dockerfile'

  buildContext: '.'

  containerRegistry: 'acrrrr.azurecr.io'

  dockerRegistryServiceConnection: 'acrserviceconnection'

  tag: 'latest2'  # Unique tag per build

pool:

  vmImage: 'ubuntu-latest'

stages:

# ========================

# 🔧 BUILD STAGE

# ========================

- stage: Build

  displayName: Build and Push Docker Image

  jobs:

  - job: Build

    displayName: Docker Build, Tag, Push

    steps:

    - script: |

        echo "Building image: $(imageName):$(tag) with platform linux/amd64"

        docker build -t $(imageName):$(tag) -f $(dockerfilePath) $(buildContext)

        docker tag $(imageName):$(tag) $(containerRegistry)/$(imageName):$(tag)

        docker images

      displayName: 'Build and Tag Docker Image'

    - task: Docker@2

      displayName: 'Push Docker Image to ACR'

      inputs:

        containerRegistry: 'ACR'

        repository: '$(imageName)'

        command: 'push'I

        tags: '$(tag)'

I am seeing following events from POD :in non working scenario


    DownwardAPI:             true

QoS Class:                   BestEffort

Node-Selectors:              <none>

Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s

                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s

Events:

  Type     Reason     Age                From               Message

  ----     ------     ----               ----               -------

  Normal   Scheduled  95s                default-scheduler  Successfully assigned default/aks-agent-deployment-5bc5ddf9b5-2tfkw to aks-np-36667834-vmss000001

  Normal   Pulling    94s                kubelet            Pulling image "acrrrr.azurecr.io/aksimage:latest2"

  Normal   Pulled     75s                kubelet            Successfully pulled image "acrrrr.azurecr.io/aksimage:latest2" in 18.364s (18.365s including waiting). Image size: 262281640 bytes.

  Normal   Created    28s (x4 over 75s)  kubelet            Created container: azp-agent

  Normal   Started    28s (x4 over 75s)  kubelet            Started container azp-agent

  Normal   Pulled     28s (x3 over 72s)  kubelet            Container image "acrrrr.azurecr.io/aksimage:latest2" already present on machine

  Warning  BackOff    0s (x6 over 71s)   kubelet            Back-off restarting failed container azp-agent in pod aks-agent-deployment-5bc5ddf9b5-2tfkw_default(9844a9b0-f1b4-4762-95f1-f846f4

Why it is failing during deployment? even both cases got pushed successfully , deployment failed in second scenario. it is same dockerfile and the code is same.

Please suggest

1 answer

Your answer

Answer 1

Vahid Ghafarpour 23,290 Moderator

It seems the container is crashing repeatedly, and likely lies within the container itself.

You can check more information using:

kubectl logs aks-agent-deployment-5bc5ddf9b5-2tfkw

Diptesh Kumar 666

HI Vahid,

1.

PS C:\Users\LAPSAdmin\AKSPool> kubectl logs aks-agent-deployment-5844cc4775-p79cc

exec ./start.sh: exec format error

I am seeing above error.

and I use below start.sh file:

FROM ubuntu:22.04
ENV TARGETARCH="linux-x64"
# Also can be "linux-arm", "linux-arm64".
# test

RUN apt update && \
  apt upgrade -y && \
  apt install -y curl git jq libicu70

# Install Azure CLI
RUN curl -sL https://aka.ms/InstallAzureCLIDeb | bash

WORKDIR /azp/

COPY ./start.sh ./
RUN chmod +x ./start.sh

# Create agent user and set up home directory
RUN useradd -m -d /home/agent agent
RUN chown -R agent:agent /azp /home/agent


# Another option is to run the agent as root.
ENV AGENT_ALLOW_RUNASROOT="true"

ENTRYPOINT [ "./start.sh" ]
shyam@DESKTOP-OO7EVK2:~/azp-agent-in-docker$ cat start.sh
#!/bin/bash
set -e

if [ -z "${AZP_URL}" ]; then
  echo 1>&2 "error: missing AZP_URL environment variable"
  exit 1
fi

if [ -n "$AZP_CLIENTID" ]; then
  echo "Using service principal credentials to get token"
  az login --allow-no-subscriptions --service-principal --username "$AZP_CLIENTID" --password "$AZP_CLIENTSECRET" --tenant "$AZP_TENANTID"
  # adapted from https://learn.microsoft.com/en-us/azure/databricks/dev-tools/user-aad-token
  AZP_TOKEN=$(az account get-access-token --query accessToken --output tsv)
  echo "Token retrieved"
fi

if [ -z "${AZP_TOKEN_FILE}" ]; then
  if [ -z "${AZP_TOKEN}" ]; then
    echo 1>&2 "error: missing AZP_TOKEN environment variable"
    exit 1
  fi

  AZP_TOKEN_FILE="/azp/.token"
  echo -n "${AZP_TOKEN}" > "${AZP_TOKEN_FILE}"
fi

unset AZP_CLIENTSECRET
unset AZP_TOKEN

if [ -n "${AZP_WORK}" ]; then
  mkdir -p "${AZP_WORK}"
fi

cleanup() {
  trap "" EXIT

  if [ -e ./config.sh ]; then
    print_header "Cleanup. Removing Azure Pipelines agent..."

    # If the agent has some running jobs, the configuration removal process will fail.
    # So, give it some time to finish the job.
    while true; do
      ./config.sh remove --unattended --auth "PAT" --token $(cat "${AZP_TOKEN_FILE}") && break

      echo "Retrying in 30 seconds..."
      sleep 30
    done
  fi
}

print_header() {
  lightcyan="\033[1;36m"
  nocolor="\033[0m"
  echo -e "\n${lightcyan}$1${nocolor}\n"
}

# Let the agent ignore the token env variables
export VSO_AGENT_IGNORE="AZP_TOKEN,AZP_TOKEN_FILE"

print_header "1. Determining matching Azure Pipelines agent..."

AZP_AGENT_PACKAGES=$(curl -LsS \
    -u user:$(cat "${AZP_TOKEN_FILE}") \
    -H "Accept:application/json" \
    "${AZP_URL}/_apis/distributedtask/packages/agent?platform=${TARGETARCH}&top=1")

AZP_AGENT_PACKAGE_LATEST_URL=$(echo "${AZP_AGENT_PACKAGES}" | jq -r ".value[0].downloadUrl")

if [ -z "${AZP_AGENT_PACKAGE_LATEST_URL}" -o "${AZP_AGENT_PACKAGE_LATEST_URL}" == "null" ]; then
  echo 1>&2 "error: could not determine a matching Azure Pipelines agent"
  echo 1>&2 "check that account "${AZP_URL}" is correct and the token is valid for that account"
  exit 1
fi

print_header "2. Downloading and extracting Azure Pipelines agent..."

curl -LsS "${AZP_AGENT_PACKAGE_LATEST_URL}" | tar -xz & wait $!

source ./env.sh

trap "cleanup; exit 0" EXIT
trap "cleanup; exit 130" INT
trap "cleanup; exit 143" TERM

print_header "3. Configuring Azure Pipelines agent..."

# Despite it saying "PAT", it can be the token through the service principal
./config.sh --unattended \
  --agent "${AZP_AGENT_NAME:-$(hostname)}" \
  --url "${AZP_URL}" \
  --auth "PAT" \
  --token $(cat "${AZP_TOKEN_FILE}") \
  --pool "${AZP_POOL:-Default}" \
  --work "${AZP_WORK:-_work}" \
  --replace \
  --acceptTeeEula & wait $!

print_header "4. Running Azure Pipelines agent..."

chmod +x ./run.sh

# To be aware of TERM and INT signals call ./run.sh
# Running it with the --once flag at the end will shut down the agent after the build is executed
./run.sh "$@" & wait $!

2.

When I pushed the image manully and when I use that image to deploy from AKS it is working well

When I pushed the image from azure pipeline and when I use that image to deploy from AKS it is failing,

it is same code and same docker file, same start.sh and same YAML file

Diptesh Kumar 666 Reputation points

2025-04-22T05:11:47.5633333+00:00

Hi Vahid,

Just checking whether did you get a chance to look my previous comment?
Arko 2,855 Reputation points Microsoft External Staff Moderator

2025-04-22T13:40:47.6733333+00:00
Hello Diptesh Kumar, Based on what you’ve shared, I believe the issue lies with a mismatch in the target platform architecture during the Docker image build inside your Azure DevOps pipeline. Your error message- exec ./start.sh: exec format error occurs when the container is trying to execute a binary or script that was built for a different CPU architecture than what the node (in your case, the AKS Linux node) supports. You need to explicitly specify the target platform during the Docker build step in your Azure pipeline.
Something like this-

- script: | echo "Building image: $(imageName):$(tag) for platform linux/amd64" docker build --platform=linux/amd64 -t $(imageName):$(tag) -f $(dockerfilePath) $(buildContext) docker tag $(imageName):$(tag) $(containerRegistry)/$(imageName):$(tag) displayName: 'Build and Tag Docker Image'

Please try and let me know. Thanks
Diptesh Kumar 666 Reputation points

2025-04-23T04:28:16.5633333+00:00

Hi Arko,

I tried above still the same behaviour and same problem.

Bheemani Anji Babu 180 Microsoft External Staff

Hi @Diptesh Kumar

could you please try the below build script:

variables:
  imageName: 'aksimage'
  dockerfilePath: 'Dockerfile'
  buildContext: '.'
  containerRegistry: 'acrrrr.azurecr.io'
  dockerRegistryServiceConnection: 'acrserviceconnection'  
  tag: 'latest2'  
pool:
  vmImage: 'ubuntu-latest'
stages:
  - stage: Build
    displayName: Build and Push Docker Image
    jobs:
      - job: Build
        displayName: Docker Build, Tag, Push
        steps:
          - script: |
              echo "Building image: $(imageName):$(tag) with platform linux/amd64"
              docker build --platform linux/amd64 -t $(imageName):$(tag) -f $(dockerfilePath) $(buildContext)
              docker tag $(imageName):$(tag) $(containerRegistry)/$(imageName):$(tag)
              docker images
            displayName: 'Build and Tag Docker Image'
          - task: Docker@2
            displayName: 'Push Docker Image to ACR'
            inputs:
              containerRegistry: $(dockerRegistryServiceConnection)  
              repository: $(imageName)  
              command: 'push'
              tags: $(tag)

you can specify this --platform linux/amd64 in the docker build command to make sure the image is built for the correct architecture,

and used the defined varible $(dockerRegistryServiceConnection) for the service connection instead of the hardcoded 'ACR'

If the updated YAML with just the image name works, it's usually the recommended method, but if there's any confusion or failure, the full registry URL ($(containerRegistry)/$(imageName)) can be specified as a fallback.

Diptesh Kumar 666 Reputation points

2025-04-26T08:39:46.9333333+00:00
Hi Bheemani Babu and Arko,

I am still seeing same error message.

node.kubernetes.io/unreachable:NoExecute op=Exists for 300s

Events:

Type Reason Age From Message

Normal Scheduled 62s default-scheduler Successfully assigned default/aks-agent-deployment-84ff6b49-4549m to aks-fd-42888116-vmss000000

Normal Pulled 32s kubelet Successfully pulled image "acrzzz.azurecr.io/aksimage:latest" in 28.643s (28.643s including waiting). Image size: 262507492 bytes.

Normal Pulled 28s kubelet Successfully pulled image "acrzzz.azurecr.io/aksimage:latest" in 248ms (248ms including waiting). Image size: 262507492 bytes.

Normal Pulling 15s (x3 over 61s) kubelet Pulling image "acrzzz.azurecr.io/aksimage:latest"

Normal Created 15s (x3 over 32s) kubelet Created container: azp-agent

Normal Started 15s (x3 over 32s) kubelet Started container azp-agent

Normal Pulled 15s kubelet Successfully pulled image "acrzzz.azurecr.io/aksimage:latest" in 195ms (195ms including waiting). Image size: 262507492 bytes.

Warning BackOff 3s (x3 over 27s) kubelet Back-off restarting failed container azp-agent in pod aks-agent-deployment-84ff6b49-4549m_default(8d9c290c-8c6c-43cd-8e1d-8844ed0a8134)
Arko 2,855 Reputation points Microsoft External Staff Moderator

2025-04-29T11:27:02.01+00:00
Hello ,

The entrypoint script start.sh is failing at runtime inside the container, because required environment variables like AZP_URL or AZP_TOKEN are missing or invalid. In your earlier logs, start.sh script has this block:

if [ -z "${AZP_URL}" ]; then echo 1>&2 "error: missing AZP_URL environment variable" exit 1 fi

and

if [ -z "${AZP_TOKEN}" ]; then echo 1>&2 "error: missing AZP_TOKEN environment variable" exit 1 fi

This means if AZP_URL is missing then container will crash immediately or if AZP_TOKEN is missing or invalid then container will crash immediately.

Please share the logs of
kubectl logs aks-agent-deployment-84ff6b49-4549m --container=azp-agent

and
kubectl describe pod aks-agent-deployment-84ff6b49-4549m

I think right now, based on the secrets you created, only AZP_TOKEN is injected from azp-secret. But if AZP_URL is not passed as an environment variable in the deployment YAML, then the container cannot connect to Azure DevOps, and it immediately crashes.

We are going step by step here troubleshooting your issue.
Originally it was platform mismatch (exec format error) which was the first blocker — we solved it.
Now the second blocker is missing environment variables causing container crash. Please check your private message for further communication on this. Thanks
Diptesh Kumar 666 Reputation points

2025-05-01T06:50:11.2966667+00:00

@Arko shared the logs and start.sh file in below ticket, hope both are describing same issue.

https://learn.microsoft.com/en-us/answers/questions/2261704/deploy-image-is-failing-in-aks-deploy-stage?comment=question&translated=false

Please assist.
Bheemani Anji Babu 180 Reputation points Microsoft External Staff

2025-05-05T06:03:07.2966667+00:00

Hi @Diptesh Kumar

I have posted an answer to your questions in the below thread link and please have a look at the answer and let us know if it helps resolve your issue. If you are still running into any problems, feel free to reach out.

https://learn.microsoft.com/en-us/answers/questions/2261535/not-able-to-deploy-aks-image-from-acr-from-pipelin?page=0&orderby=Helpful&translated=false#answers

Share via

FDeploy Image is failing - AKS

1 answer

Your answer