Issues with CI/CD Setup on Azure with VM, ACR, and GitHub (Docker Compose + Traefik Reverse Proxy)

Question

Hello,

I'm setting up a CI/CD pipeline using Azure DevOps, a Virtual Machine (VM) on Azure, and Azure Container Registry (ACR). My project uses Docker Compose and Traefik as a reverse proxy. Due to specific requirements, I'm using a VM instead of a managed service.

However, I'm encountering an issue where my VM isn't pulling images from ACR correctly. It seems like there might be a problem with the image tags or the way the images are being referenced.

Here’s the error I’m receiving:

2024-08-14T10:40:44.7501592Z ##[error]Pulling ai_backend (***/ai_backend:latest)...

2024-08-14T10:40:44.9192452Z ##[error]manifest for ***/ai_backend:latest not found: manifest unknown: manifest tagged by "latest" is not found

2024-08-14T10:40:45.0897108Z ##[section]Finishing: SSH

It looks like the 'latest' tag might not be found, but I'm not sure if this is a configuration issue with the VM, ACR, or the CI/CD pipeline itself.

Has anyone encountered a similar issue or can provide guidance on what might be going wrong?

Below are the relevant files:

• azure-pipelines.yml
• docker-compose.yml

Any help would be greatly appreciated!

Please note that my project is based on the Full-Stack FastAPI Template available on GitHub.

azure-pipelines.yml:

trigger:

- main

resources:

- repo: self

variables:

  vmImageName: 'ubuntu-latest'

  group: 'XXXXXXXXXXX'

stages:

- stage: Build

  displayName: Build and deploy stage

  jobs:

  - job: BuildAndDeploy

    displayName: Build and Deploy

    pool:

      vmImage: $(vmImageName)

    steps:

    - task: AzureCLI@2

      inputs:

        azureSubscription: 'XXXXXXXXXXX (XXXXXXXXXXX)'

        scriptType: bash

        scriptLocation: inlineScript

        inlineScript: |

          az acr login --name markingsystemreg

    - task: Docker@2

      inputs:

        command: login

        containerRegistry: 'markingsystemreg'

    - task: Docker@2

      displayName: Build and Push ai_backend image

      inputs:

        command: buildAndPush

        dockerfile: 'ai_backend/Dockerfile'

        repository: 'markingsystemreg.azurecr.io/ai_backend'

        tags: |

          $(Build.BuildId)

          latest

    - task: Docker@2

      displayName: Build and Push backend image

      inputs:

        command: buildAndPush

        dockerfile: 'backend/Dockerfile'

        repository: 'markingsystemreg.azurecr.io/backend'

        tags: |

          $(Build.BuildId)

          latest

    - task: Docker@2

      displayName: Build and Push frontend image

      inputs:

        command: buildAndPush

        dockerfile: 'frontend/Dockerfile'

        repository: 'markingsystemreg.azurecr.io/frontend'

        tags: |

          $(Build.BuildId)

          latest

    - task: CopyFilesOverSSH@0

      inputs:

        sshEndpoint: 'my-ssh-connection'

        sourceFolder: '$(System.DefaultWorkingDirectory)'

        targetFolder: '/home/XXXXXXXXXXX/deploy'

        cleanTargetFolder: true

    - task: SSH@0

      inputs:

        sshEndpoint: 'my-ssh-connection'

        runOptions: inline

        inline: |

          clientId="XXXXXXXXXXX"

          clientSecret="XXXXXXXXXXX"

          tenantId="XXXXXXXXXXX"

          # Install Azure CLI

          curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash

          az --version

          # Login to Azure

          az login --service-principal -u $clientId -p $clientSecret --tenant $tenantId

          az acr login --name markingsystemreg

          # Get the latest tag from the repository

          LATEST_TAG=$(az acr repository show-tags --name markingsystemreg --repository markingsystemreg.azurecr.io/ai_backend --output tsv --orderby time_desc --top 1)

          

          # Navigate to the deployment directory

          cd /home/XXXXXXXXXXX/deploy

          # Install Docker Compose

          sudo curl -sL "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose

          sudo chmod +x /usr/local/bin/docker-compose

          docker-compose --version

          # Update the docker-compose.yml to use the correct tag

          sed -i "s/:latest/:$LATEST_TAG/" docker-compose.yml

    - task: SSH@0

      inputs:

        sshEndpoint: 'my-ssh-connection'

        runOptions: inline

        inline: |

          az acr login --name markingsystemreg

          cd /home/AssessmentProjectVMUser/deploy

          docker-compose -f docker-compose.yml up -d

docker-compose.yml:

version: '3.8'

services:

  rabbitmq:

    image: rabbitmq:3.13-management

    container_name: rabbitmq

    hostname: rabbitmq

    ports:

      - "5672:5672"

      - "15672:15672"

    environment:

      - RABBITMQ_DEFAULT_USER=${RABBITMQ_DEFAULT_USER}

      - RABBITMQ_DEFAULT_PASS=${RABBITMQ_DEFAULT_PASS}

    restart: unless-stopped

    networks:

      - default

    healthcheck:

      test: ["CMD-SHELL", "rabbitmqctl status"]

      interval: 10s

      timeout: 5s

      retries: 5

  ai_backend:

    image: markingsystemreg.azurecr.io/ai_backend:${TAG:-latest}

    restart: always

    depends_on:

      - rabbitmq

    env_file:

      - .env

    environment:

      - PYTHONPATH=/app/ai_backend

    command: ["./wait-for-it.sh", "rabbitmq:5672", "--timeout=60", "--", "uvicorn", "api:app", "--host", "0.0.0.0", "--port", "85"]

    networks:

      - default

  pdf_consumer:

    build:

      context: ./ai_backend

      dockerfile: Dockerfile

      target: pdf_consumer

    depends_on:

      - rabbitmq

    env_file:

      - .env

    environment:

      - PYTHONPATH=/app/ai_backend

    command: ["./wait-for-it.sh", "rabbitmq:5672", "--timeout=60", "--", "python", "RMQ/consumers/pdf_process_consumer.py"]

    networks:

      - default

  adminer:

    image: adminer

    restart: always

    networks:

      - traefik-public

      - default

    environment:

      - ADMINER_DESIGN=pepa-linha-dark

    labels:

      - traefik.enable=true

      - traefik.docker.network=traefik-public

      - traefik.constraint-label=traefik-public

      - traefik.http.routers.${STACK_NAME?Variable not set}-adminer-http.rule=Host(`adminer.${DOMAIN?Variable not set}`)

      - traefik.http.routers.${STACK_NAME?Variable not set}-adminer-http.entrypoints=http

      - traefik.http.routers.${STACK_NAME?Variable not set}-adminer-http.middlewares=https-redirect

      - traefik.http.routers.${STACK_NAME?Variable not set}-adminer-https.rule=Host(`adminer.${DOMAIN?Variable not set}`)

      - traefik.http.routers.${STACK_NAME?Variable not set}-adminer-https.entrypoints=https

      - traefik.http.routers.${STACK_NAME?Variable not set}-adminer-https.tls=true

      - traefik.http.routers.${STACK_NAME?Variable not set}-adminer-https.tls.certresolver=le

      - traefik.http.services.${STACK_NAME?Variable not set}-adminer.loadbalancer.server.port=8080

  backend:

    image: markingsystemreg.azurecr.io/backend:${TAG:-latest}

    restart: always

    networks:

      - traefik-public

      - default

    env_file:

      - .env

    environment:

      - DOMAIN=${DOMAIN}

      - ENVIRONMENT=${ENVIRONMENT}

      - BACKEND_CORS_ORIGINS=${BACKEND_CORS_ORIGINS}

      - SECRET_KEY=${SECRET_KEY?Variable not set}

      - FIRST_SUPERUSER=${FIRST_SUPERUSER?Variable not set}

      - FIRST_SUPERUSER_PASSWORD=${FIRST_SUPERUSER_PASSWORD?Variable not set}

      - USERS_OPEN_REGISTRATION=${USERS_OPEN_REGISTRATION}

      - SMTP_HOST=${SMTP_HOST}

      - SMTP_USER=${SMTP_USER}

      - SMTP_PASSWORD=${SMTP_PASSWORD}

      - EMAILS_FROM_EMAIL=${EMAILS_FROM_EMAIL}

      - AZURE_SQL_SERVER=${AZURE_SQL_SERVER}

      - AZURE_SQL_PORT=${AZURE_SQL_PORT}

      - AZURE_SQL_DATABASE=${AZURE_SQL_DATABASE}

      - AZURE_SQL_USERNAME=${AZURE_SQL_USERNAME}

      - AZURE_SQL_PASSWORD=${AZURE_SQL_PASSWORD}

      - AZURE_SQL_DRIVER=${AZURE_SQL_DRIVER}

      - AZURE_SQL_ENCRYPT=${AZURE_SQL_ENCRYPT}

      - AZURE_SQL_TRUST_CERTIFICATE=${AZURE_SQL_TRUST_CERTIFICATE}

      - TIMEOUT=${TIMEOUT}

      - SENTRY_DSN=${SENTRY_DSN}

    build:

      context: ./backend

      args:

        INSTALL_DEV: ${INSTALL_DEV-false}

    labels:

      - traefik.enable=true

      - traefik.docker.network=traefik-public

      - traefik.constraint-label=traefik-public

      - traefik.http.services.${STACK_NAME?Variable not set}-backend.loadbalancer.server.port=80

      - traefik.http.routers.${STACK_NAME?Variable not set}-backend-http.rule=Host(`${DOMAIN?Variable not set}`, `www.${DOMAIN?Variable not set}`) && PathPrefix(`/api`, `/docs`, `/redoc`)

      - traefik.http.routers.${STACK_NAME?Variable not set}-backend-http.entrypoints=http

      - traefik.http.routers.${STACK_NAME?Variable not set}-backend-https.rule=Host(`${DOMAIN?Variable not set}`, `www.${DOMAIN?Variable not set}`) && PathPrefix(`/api`, `/docs`, `/redoc`)

      - traefik.http.routers.${STACK_NAME?Variable not set}-backend-https.entrypoints=https

      - traefik.http.routers.${STACK_NAME?Variable not set}-backend-https.tls=true

      - traefik.http.routers.${STACK_NAME?Variable not set}-backend-https.tls.certresolver=le

      - traefik.http.routers.${STACK_NAME?Variable not set}-backend-http.middlewares=https-redirect,${STACK_NAME?Variable not set}-www-redirect

      - traefik.http.routers.${STACK_NAME?Variable not set}-backend-https.middlewares=${STACK_NAME?Variable not set}-www-redirect

  frontend:

    image: markingsystemreg.azurecr.io/frontend:${TAG:-latest}

    restart: always

    networks:

      - traefik-public

      - default

    build:

      context: ./frontend

      args:

        - VITE_API_URL=https://${DOMAIN?Variable not set}

        - NODE_ENV=production

    labels:

      - traefik.enable=true

      - traefik.docker.network=traefik-public

      - traefik.constraint-label=traefik-public

      - traefik.http.services.${STACK_NAME?Variable not set}-frontend.loadbalancer.server.port=80

      - traefik.http.routers.${STACK_NAME?Variable not set}-frontend-http.rule=Host(`${DOMAIN?Variable not set}`, `www.${DOMAIN?Variable not set}`)

      - traefik.http.routers.${STACK_NAME?Variable not set}-frontend-http.entrypoints=http

      - traefik.http.routers.${STACK_NAME?Variable not set}-frontend-https.rule=Host(`${DOMAIN?Variable not set}`, `www.${DOMAIN?Variable not set}`)

      - traefik.http.routers.${STACK_NAME?Variable not set}-frontend-https.entrypoints=https

      - traefik.http.routers.${STACK_NAME?Variable not set}-frontend-https.tls=true

      - traefik.http.routers.${STACK_NAME?Variable not set}-frontend-https.tls.certresolver=le

      - traefik.http.middlewares.${STACK_NAME?Variable not set}-www-redirect.redirectregex.regex=^http(s)?://www.(${DOMAIN?Variable not set})/(.*)

      - traefik.http.middlewares.${STACK_NAME?Variable not set}-www-redirect.redirectregex.replacement=http$${1}://${DOMAIN?Variable not set}/$${3}

      - traefik.http.routers.${STACK_NAME?Variable not set}-frontend-https.middlewares=${STACK_NAME?Variable not set}-www-redirect

      - traefik.http.routers.${STACK_NAME?Variable not set}-frontend-http.middlewares=https-redirect,${STACK_NAME?Variable not set}-www-redirect

networks:

  traefik-public:

    external: true

As you can see from the screenshot when I initially listed the available images, it displayed ones that were created 26 hours ago.

Next, I manually pulled the ai_backend Docker image, which had actually been updated two hours ago with Azure DevOps. Once manual pull process finished, I listed images again and it showed as being created two hours ago. This suggests that I may not be pulling the images properly.

Additionally, even when I tried running docker-compose manually on the VM, I still encountered the same error.

Answer 1

We do not currently provide support for Azure DevOps on this site. Please refer to their official support channel to get assistance with setting up a CI/CD pipeline.