Error on Azure IoT Edge Task in Release pipeline for docker hub image pull

Dhanavath Vishnu 366

Hi Team

I am facing an issue in my standard IoT edge deployment template in the release pipeline. In the template I have declared mongo dB has one of the modules, the image URL I am using his: docker.io/library/mongo:4.4.18. While I am doing the deployment manually using the portal, it is able to successfully pull the image deploy on the virtual machine.

But while I use the same template to generate from the pipeline, during the deployment to the device using Azure IoT Edge Task, I am getting the following error.

User's image

As I am not having any docker credentials while doing from the portal, should I need to pass the docker registry details in the template file so this error will go away. As I see the image is public, without docker credentials also it should work right?

Please clarify, if any docs to refer please do let me know.

D. Vishnu

LeelaRajeshSayana-MSFT 16,701 Reputation points

2023-05-02T16:49:39.8566667+00:00

Hi @Dhanavath Vishnu Greetings! I was referring to the Azure IoT Edge v2 task variable template syntax and noticed that there is place holder variable containerregistrytype which points to the container for the image. The document states the default value for this is Azure Container Registry

Can you confirm if you are setting the value to this parameter as Generic Container Registry and providing the registry URL to the variable dockerRegistryConnection and still experience this issue?
VasimTamboli 5,095 Reputation points

2023-05-02T18:28:06.4066667+00:00

Hello @Dhanavath Vishnu,

Based on the error message in the screenshot you shared, it seems that the Azure IoT Edge Task is not able to authenticate to the Docker Hub registry to pull the mongo:4.4.18 image.

If the mongo image is a public image on Docker Hub, you do not need any docker registry details or credentials to pull the image. However, if the image is private, you need to provide the Docker Hub registry credentials in the template file.

To resolve the issue, you can try the following:

Check if the mongo:4.4.18 image is public or private on Docker Hub. If it is private, you need to provide the Docker Hub registry credentials in the template file.

Check if the Azure IoT Edge Task is configured to use the correct Docker Hub registry URL. The URL should be docker.io for Docker Hub.

Check if the Azure IoT Edge Task is configured to use the correct Docker Hub registry credentials, if required.

If you have configured Docker Hub registry credentials in the Azure IoT Edge Task, make sure they are correct and have sufficient permissions to pull the mongo:4.4.18 image.

You can also try using the fully qualified image name, docker.io/library/mongo:4.4.18, instead of just mongo:4.4.18.

You can refer to the Azure IoT Edge documentation for more information on configuring Docker registry credentials in Azure IoT Edge tasks.
Nicolas Christiaens 15 Reputation points

2023-05-03T06:11:55.41+00:00

I'm experiencing exactly the same issue, my pipeline hasn't changed in months, my deployment file is still the same. Everything worked fine a few weeks ago but yesterday I got exactly the same error.

I am using public images on docker.io and I am using the fully qualified image name (and the actual deployment target edge server can pull the images correctly with docker pull). Any tips on how to proceed would be highly welcome, our testing has come to a halt because we can't deploy!

I haven't changed the container registry type as suggested above, because we use both ACR and docker.io, only the docker.io entries seem to be faulting.

Thanks!
LeelaRajeshSayana-MSFT 16,701 Reputation points

2023-05-03T21:30:45.99+00:00

We are checking on this issue internally. We will get back to you with more details as soon as possible. Thank you for your patience. We apologize for the inconvenience this is causing.
LeelaRajeshSayana-MSFT 16,701 Reputation points

2023-05-03T23:46:35.45+00:00

I would like to update that our team can replicate the same behavior internally. Would it be possible to get logs from the successful runs so that we can compare which dependencies have changed versions? Identifying the exact versions will help us pin them more strictly in the toolset.

I appreciate it if you can share the logs to the email azcommunity@microsoft.com with the below details, so that we can work closely on this matter.

Thread URL: Link to this thread.
Email Subject : Attn Leela
LeelaRajeshSayana-MSFT 16,701 Reputation points

2023-05-05T12:44:01.4133333+00:00

Hi @Dhanavath Vishnu I see that you are encountering an error during image pull as well. If the below work around proposed does not work for you, I appreciate it if you can share logs from a successful run to the email azcommunity@microsoft.com for further investigation.
LeelaRajeshSayana-MSFT 16,701 Reputation points

2023-05-11T13:22:29.5733333+00:00
Hi @Dhanavath Vishnu We have an update from the product team to address the credential issue. Here are the steps to be followed to resolve this.

Replace Azure IoT Edge - Deploy to IoT Edge devices job with this Azure CLI job. Both are equivalent and can be used interchangeably.

- task: AzureCLI@2 displayName: 'Azure CLI ' inputs: azureSubscription: '<Azure-Resource-Manager-service-connection-name>' scriptType: bash scriptLocation: inlineScript inlineScript: | az extension add --name azure-iot

Delete any existing deployments and create a new deployment. Please give this a try and let us know if you still encounter any issues. Looking forward to your feedback.

I have updated the answer below to reflect the same. If the response helped, please do click Accept Answer and Yes for the answer provided. Doing so would help other community members with similar issue identify the solution. I highly appreciate your contribution to the community.
LeelaRajeshSayana-MSFT 16,701 Reputation points

2023-05-15T15:53:49.24+00:00

Hi @Dhanavath Vishnu Greetings! I am following up to see if you got a chance to review the below response. Please let us know if you have any additional questions or need further assistance.
LeelaRajeshSayana-MSFT 16,701 Reputation points

2023-05-18T14:04:06.8933333+00:00

Hi @Dhanavath Vishnu Greetings! We still have not heard back from you. Just wanted to check if the answer provided below was helpful? If it answers your query, please do click Accept Answer and Yes for the answer, as it might be beneficial to other community members reading this thread. And, if you have any further query do let us know. Thank you

Accepted answer

LeelaRajeshSayana-MSFT 16,701

@Dhanavath Vishnu We have update from our team on this issue.

Based on the logs the team has analyzed so far, the root cause of the issue is due to the docker CLI failing to authenticate. This issue can manifest itself in either the 'push' action or the 'deploy' action of the AzureIoTEdgeV2 devops task.

The team is still working on a permanent fix. To mitigate this in the meantime, please follow the below steps.

For push action:

- task: Docker@2
  displayName: push
  inputs:
    containerRegistry: <Docker-registry-service-connection-name>
    repository: '<DockerHub-repository-name>'
    command: push
    tags: '<module-version>-<module-arch>' #ex: 0.0.1-amd64

For deploy action:

- task: AzureCLI@2
  displayName: 'Azure CLI '
  inputs:
    azureSubscription: '<Azure-Resource-Manager-service-connection-name>'
    scriptType: bash
    scriptLocation: inlineScript
    inlineScript: |
     az extension add --name azure-iot

Please make sure to delete any previous deployments if they exist and create a new deployment using the following steps.

Update to address docker login error

Replace Azure IoT Edge - Deploy to IoT Edge devices job with this Azure CLI job. Both are equivalent and can be used interchangeably.

- task: AzureCLI@2
  displayName: 'Azure CLI '
  inputs:
    azureSubscription: '<Azure-Resource-Manager-service-connection-name>'
    scriptType: bash
    scriptLocation: inlineScript
    inlineScript: |
     az extension add --name azure-iot

Please ensure to delete any existing deployments and create new one using the steps below:

# Delete previous deployment, if one exists      
project=$(echo $(System.TeamProject) | tr [:upper:] [:lower:])      
az iot edge deployment delete --deployment-id $project-devops-deployment --hub-name <IoTHub-name> || true       
# Create a new deployment      
az iot edge deployment create --deployment-id $(System.TeamProject)-devops-deployment --hub-name <IoTHub-name> --content $(System.DefaultWorkingDirectory)/config/deployment.json --target-condition "deviceId='<device-name>'" --priority 10 --output none

Please let us know if you encounter any issues or need further clarifications on the above.

If the response helped, please do click Accept Answer and Yes. Doing so would help other community members with similar issue identify the solution. I highly appreciate your contribution to the community.

Nicolas Christiaens 15

Hi Leela,

I've tried this, however I still get the same error. I don't see why the docker cli should authenticate for public docker images? If I use the URL myself, I also get an unauthorized exception. Isn't the deploy step just somehow getting a wrong URL?

For example: https://registry-1.docker.io/v2/library/rabbitmq/manifests/3.10.7-alpine

In my deployment manifest I have the following image specified: "image": "docker.io/rabbitmq:3.10.7-alpine"

The following command executes perfectly: docker pull docker.io/rabbitmq:3.10.7-alpine

Even the command that fails during the deployment step works perfectly on my local development PC:

Command:

$ docker manifest inspect docker.io/rabbitmq:3.10.7-alpine


{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 2200,
         "digest": "sha256:5797a921afe26233e980a51bc0db22dd1701d02dc4671f74fca2903d9cc57842",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 2200,
         "digest": "sha256:9ce7eaf8c40a5a04c94217109c8bba58c4b5ced49121bc57cbcd463ad44c5eca",
         "platform": {
            "architecture": "arm",
            "os": "linux",
            "variant": "v6"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 2200,
         "digest": "sha256:54b35fc3d9ce86121ce2b1a7fd4ad79fcc71b2929b9a287a299d84664e5ce584",
         "platform": {
            "architecture": "arm",
            "os": "linux",
            "variant": "v7"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 2200,
         "digest": "sha256:8b5eef649087159d1883c14953bb23af14b291cdc11757b1d4e3463af5026535",
         "platform": {
            "architecture": "arm64",
            "os": "linux",
            "variant": "v8"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 2200,
         "digest": "sha256:bc9af0c0ff31c212c60436462513c464cdbf67c541382c6c8f90de8120ed7164",
         "platform": {
            "architecture": "386",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 2200,
         "digest": "sha256:c338f1235d2c70b0d46ed2c6c510f216aa70c7856d0456b93bdfa39c81ee3f85",
         "platform": {
            "architecture": "ppc64le",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 2200,
         "digest": "sha256:d50390e4b3ca984e8289e2b8b7504f0ec26718c983547d0b7cfe709952267d22",
         "platform": {
            "architecture": "s390x",
            "os": "linux"
         }
      }
   ]
}

LeelaRajeshSayana-MSFT 16,701 Reputation points

2023-05-09T18:16:46.1566667+00:00

Hi Nicolas, Thank you for your feedback. We are reviewing this internally with the team. I will keep you posted with more details. Please stay tuned.

Share via

Error on Azure IoT Edge Task in Release pipeline for docker hub image pull

0 additional answers

Your answer