Assistance Required for Docker Signing on Azure Hosted Agents

Question

Hi Team,
I am currently working on implementing Docker signing in our Azure DevOps pipeline using Azure hosted agents. However, I have encountered some issues and would appreciate your guidance.

I have followed the official Microsoft documentation on this subject: Content Trust in Azure DevOps. Despite adhering to the guidelines, the pipeline is not functioning as expected.

Could you please assist me in identifying any potential issues with the configuration or suggest any additional steps I might need to take? I have attached the pipeline configuration file for your reference.

pool:
  vmImage: 'ubuntu-latest'

variables:
  system.debug: true
  containerRegistryServiceConnection: test
  imageRepository: testy.azurecr.io/datacollectionandvalidationbackend4
  tag: test
  DOCKER_CONTENT_TRUST_REPOSITORY_PASSPHRASE: 'ttt@123'

steps:
- task: Docker@2
  inputs:
    command: login
    containerRegistry: $(containerRegistryServiceConnection)

# Download and configure signing keys
- task: DownloadSecureFile@1
  name: privateKey
  inputs:
    secureFile: 12f461684ec131fd7ee3f58d8f11ceb7de184e76359780ed26fc54c601b90925.key
- script: |
    mkdir -p $(DOCKER_CONFIG)/trust/private
    cp $(privateKey.secureFilePath) $(DOCKER_CONFIG)/trust/private
    ls $(DOCKER_CONFIG)/trust/private

- task: DownloadSecureFile@1
  name: privateKey1
  inputs:
    secureFile: ce38f605160341b4b96737321d08c0020744140ebfb11dc40e44f0fe5ead2a1a.key
- script: |
    mkdir -p $(DOCKER_CO NFIG)/trust/private
    cp $(privateKey1.secureFilePath) $(DOCKER_CONFIG)/trust/private
    ls $(DOCKER_CONFIG)/trust/private

- task: DownloadSecureFile@1
  name: privateKey2
  inputs:
    secureFile: fa72538d0766d0cf99bcab2c0d68dfb1edbdf3610950e9061780c1d3b8e3420c.key
- script: |
    mkdir -p $(DOCKER_CONFIG)/trust/private
    cp $(privateKey2.secureFilePath) $(DOCKER_CONFIG)/trust/private
    ls $(DOCKER_CONFIG)/trust/private

# Build Docker image with Content Trust enabled
- task: Docker@2
  inputs:
    command: build
    Dockerfile: '**/Dockerfile'
    containerRegistry: $(containerRegistryServiceConnection)
    repository: $(imageRepository)
    tags: |
      $(tag)
  env:
    DOCKER_CONTENT_TRUST: 1

# Push Docker image with Content Trust enabled
- task: Docker@2
  inputs: 
    command: push
    containerRegistry: $(containerRegistryServiceConnection)
    repository: $(imageRepository)
    tags: |
      $(tag)
  env:
    DOCKER_CONTENT_TRUST: 1
    DOCKER_CONTENT_TRUST_REPOSITORY_PASSPHRASE: $(DOCKER_CONTENT_TRUST_REPOSITORY_PASSPHRASE)

And there is no error the pipeline keep running with the below message
##[debug]Agent environment resources - Disk: / Available 19769.33 MB out of 74244.74 MB, Memory: Used 801.00 MB out of 6921.00 MB, CPU: Usage 2.72%

##[debug]Agent environment resources - Disk: / Available 19769.32 MB out of 74244.74 MB, Memory: Used 801.00 MB out of 6921.00 MB, CPU: Usage 2.72%

##[debug]Agent environment resources - Disk: / Available 19769.32 MB out of 74244.74 MB, Memory: Used 801.00 MB out of 6921.00 MB, CPU: Usage 2.72%

##[debug]Agent environment resources - Disk: / Available 19769.32 MB out of 74244.74 MB, Memory: Used 801.00 MB out of 6921.00 MB, CPU: Usage 2.71%

##[debug]Agent environment resources - Disk: / Available 19769.31 MB out of 74244.74 MB, Memory: Used 801.00 MB out of 6921.00 MB, CPU: Usage 2.71%

##[debug]Agent environment resources - Disk: / Available 19769.30 MB out of 74244.74 MB, Memory: Used 801.00 MB out of 6921.00 MB, CPU: Usage 2.71%

##[debug]Agent environment resources - Disk: / Available 19769.30 MB out of 74244.74 MB, Memory: Used 801.00 MB out of 6921.00 MB, CPU: Usage 2.71%

##[debug]Agent environment resources - Disk: / Available 19769.30 MB out of 74244.74 MB, Memory: Used 801.00 MB out of 6921.00 MB, CPU: Usage 2.70%

##[debug]Agent environment resources - Disk: / Available 19769.30 MB out of 74244.74 MB, Memory: Used 801.00 MB out of 6921.00 MB, CPU: Usage 2.70%

##[debug]Agent environment resources - Disk: / Available 19769.30 MB out of 74244.74 MB, Memory: Used 801.00 MB out of 6921.00 MB, CPU: Usage 2.70%

##[debug]Agent environment resources - Disk: / Available 19769.29 MB out of 74244.74 MB, Memory: Used 801.00 MB out of 6921.00 MB, CPU: Usage 2.70%

##[debug]Agent environment resources - Disk: / Available 19769.29 MB out of 74244.74 MB, Memory: Used 801.00 MB out of 6921.00 MB, CPU: Usage 2.69%

##[debug]Agent environment resources - Disk: / Available 19769.28 MB out of 74244.74 MB, Memory: Used 801.00 MB out of 6921.00 MB, CPU: Usage 2.69%

##[debug]Agent environment resources - Disk: / Available 19769.28 MB out of 74244.74 MB, Memory: Used 801.00 MB out of 6921.00 MB, CPU: Usage 2.69%

##[debug]Agent environment resources - Disk: / Available 19769.27 MB out of 74244.74 MB, Memory: Used 801.00 MB out of 6921.00 MB, CPU: Usage 2.69%

##[debug]Agent environment resources - Disk: / Available 19769.27 MB out of 74244.74 MB, Memory: Used 801.00 MB out of 6921.00 MB, CPU: Usage 2.68%

##[debug]Agent environment resources - Disk: / Available 19769.27 MB out of 74244.74 MB, Memory: Used 801.00 MB out of 6921.00 MB, CPU: Usage 2.68%

##[debug]Agent environment resources - Disk: / Available

Answer 1

Hello Shiva Kumar,

Welcome to the Microsoft Q&A and thank you for posting your questions here.

Problem

I understand that your Azure DevOps pipeline is not functioning correctly and is stuck at a stage where it repeatedly displays debug messages indicating resource availability.

Solution

To address the issues with Docker signing in your Azure DevOps pipeline, there can be several approaches but here I provide through a detailed troubleshooting approach.

1. Ensure that the necessary environment variables are set correctly in your configuration. These variables control Docker Content Trust: Though your DOCKER_CONTENT_TRUST: Is Set this to 1 to enable content trust. But ensure DOCKER_CONTENT_TRUST_REPOSITORY_PASSPHRASE: This should contain the passphrase for the Notary signing keys. Ensure this value is securely stored and retrieved.

      variables:
        DOCKER_CONTENT_TRUST: 1
        DOCKER_CONTENT_TRUST_REPOSITORY_PASSPHRASE: 'your-passphrase'
      
      
   

   1. Ensure the secure files (private keys) are correctly downloaded and placed in the Docker configuration directory. Each key should be correctly named and located. For example,
   - task: DownloadSecureFile@1 name: privateKey inputs: secureFile: <secureFileID> - script: | mkdir -p $(DOCKER_CONFIG)/trust/private cp $(privateKey.secureFilePath) $(DOCKER_CONFIG)/trust/private echo "Key copied to Docker trust directory" ls -la $(DOCKER_CONFIG)/trust/private displayName: 'Setup Docker Trust Keys' Repeat the DownloadSecureFile and script steps for each private key you need.
   1. Add detailed logging to each step to understand where the process might be hanging. This can involve simple echo commands or checking file and directory contents. For example,

      - script: |
          echo "Creating Docker trust directory at $(DOCKER_CONFIG)/trust/private"
          mkdir -p $(DOCKER_CONFIG)/trust/private
          echo "Copying signing key from $(privateKey.secureFilePath)"
          cp $(privateKey.secureFilePath) $(DOCKER_CONFIG)/trust/private
          echo "Listing contents of trust directory:"
          ls -la $(DOCKER_CONFIG)/trust/private
        displayName: 'Setup Docker Trust with Debugging'
   

   1. Ensure that the Docker build and push commands are configured correctly, including using the correct repository and tag. For example, build and push configuration:

      - task: Docker@2
        inputs:
          command: build
          Dockerfile: '**/Dockerfile'
          containerRegistry: $(containerRegistryServiceConnection)
          repository: $(imageRepository)
          tags: |
            $(tag)
          arguments: --progress=plain
        env:
          DOCKER_CONTENT_TRUST: 1
        displayName: 'Docker Build with Content Trust'
      - task: Docker@2
        inputs:
          command: push
          containerRegistry: $(containerRegistryServiceConnection)
          repository: $(imageRepository)
          tags: |
            $(tag)
        env:
          DOCKER_CONTENT_TRUST: 1
          DOCKER_CONTENT_TRUST_REPOSITORY_PASSPHRASE: $(DOCKER_CONTENT_TRUST_REPOSITORY_PASSPHRASE)
        displayName: 'Docker Push with Content Trust'
   

   The --progress=plain argument provides detailed output during the build process, which can help in identifying issues.
2. In cases where network issues or other transient problems might be causing hangs, consider implementing timeouts and retries. For example,
   • Add a timeout to script steps:

        - script: |
            # Your script here
          timeoutInMinutes: 15
   

   • Implement retry logic if supported by the tasks or manually in scripts.

Accept Answer

I hope this is helpful! Do not hesitate to let me know if you have any other questions.

** Please don't forget to close up the thread here by upvoting and accept it as an answer if it is helpful ** so that others in the community facing similar issues can easily find the solution.

Best Regards,

Sina Salam

Answer 2

Actually it was not working in azure microsoft hosted machine. I used the self hosted machine it worked. Thanks