Unable to pull remote image for multi-container web application ("redis:latest")

Nycticorax 16

Fresh from Deployment Center > Logs:

2022-06-09T09:22:09.025Z INFO - Pulling image: redis:latest
2022-06-09T09:22:09.708Z ERROR - DockerApiException: Docker API responded with status code=InternalServerError, response={"message":"Get https://registry-1.docker.io/v2/library/redis/manifests/latest: unauthorized: incorrect username or password"}

2022-06-09T09:22:09.708Z ERROR - Pulling docker image redis:latest failed:
2022-06-09T09:22:09.708Z WARN - Image pull failed. Defaulting to local copy if present.

The web app using this deployment setup has been working flawlessly for several months (it's a multi-container web application deployed from my Azure Container Registry (via webhook; see screenshot for more details). Not sure what to do to troubleshoot and fix the issue.

Ryan Hill 25,666 Reputation points Microsoft Employee

2022-06-09T18:02:51.147+00:00

When exactly did the issue begin to occur @Nycticorax ? What region is your app located in?
Nycticorax 16 Reputation points

2022-06-09T20:03:27.453+00:00
When: I noticed the issue (which might have occurred earlier) today, 9th of June 2022. I suspect it happened between the 7th and the 9th because I would have noticed it from my CD/CI routine (it would have disrupted it).

Where: Western Europe

Also I found a workaround: instead of declaring from my docker-compose:

image: redis:latest

I am declaring:

image: <my-ACR/my-image:version>
after pulling the said image on my local machine and pushing it there.

So there's definitely something fishy with the way the Web App Service fetches remote images from DockerHub.io.

If you have a better solution I am all ears. It's sad not to be able to pull from DockerHub.
Ryan Hill 25,666 Reputation points Microsoft Employee

2022-06-09T20:31:06.99+00:00

@Tobias Weber reported the same issue, I'm checking with the product group to see what's going on.
Nycticorax 16 Reputation points

2022-06-13T07:13:02.457+00:00

Fine, keep us posted if you can!
MODwyer 1 Reputation point

2022-06-17T06:11:26.247+00:00

We're seeing a similar issue with a multicontainer app that pulls one container from azurecr private registry and then reis from docker.
We tested changing the app to just pull the redis image from docker, with the source of "Docker Hub" rather than "Azure Container Registry" and it was able to pull fine. So it seems that the custom configuration settings for "DOCKER_REGISTRY_SERVER_*" that are configured for the azurecr might now be getting passed on to Docker registry too, and obviously fail due to them being invalid.

Currently working around in the same way of importing the container image directly into the private azurecr.
Ryan Hill 25,666 Reputation points Microsoft Employee

2022-06-17T20:06:23.327+00:00

@Nycticorax , I see that you're using admin credentials with ACR. @Tobias Weber , @MODwyer are you both also using Admin credentials or Managed Identity? Also, does @Nycticorax workaround using ACR for all images an effective workaround?

Can each of you send me the name of your app service along with a link to this thread to AzCommunity[at]microsoft[dot]com ATTN: Ryan so I can check a couple of things. Thanks in advance.

1 answer

Ryan Hill 25,666 Reputation points Microsoft Employee

2022-06-30T16:23:45.61+00:00

Apologies to all for the delay.

There was a recent change for security reasons that removed an unauthenticated fallback for docker pulls from docker hub. For anyone impacted by this, you can mitigate by moving the affected image to your Azure Container Registry and then adding DOCKER_REGISTRY_SERVER_USERNAME and DOCKER_REGISTRY_SERVER_PASSWORD to your app settings.

Adding no auth option is being worked on; however, there isn't an ETA on when it will be available at this time.
Please sign in to rate this answer.
Anselm Coogan 1 Reputation point

2022-08-24T17:39:16.42+00:00

Finally! Would be good if these types of changes that break working systems wouldn't "just happen" w/o any communication and paying customers need to search for this very well hidden solution.
Sign in to comment