Unable to import image to container registry

Question

Unable to import image to container registry

AgBe DevOps 1

Hi,

I have been using import operation in Azure Container Registry for some time without any issues but lately I am getting the below error consistently.

$ az acr import --name *****registry --source quay.io/jetstack/cert-manager-controller:v1.5.4 --image jetstack/cert-manager-controller:v1.5.4

(InvalidParameters) Operation registries-6ae76c3e-806e-11ed-94b2-#### failed. Resource /subscriptions/b38946e9-e907-4201-a5e6-####/resourceGroups/####-rg/providers/Microsoft.ContainerRegistry/registries/*****registry Error copying blobs. Error copying blobs. Error copying blobs. Error copying blobs. Error copying blobs.
Code: InvalidParameters
Message: Operation registries-6ae76c3e-806e-11ed-94b2-#### failed. Resource /subscriptions/b38946e9-e907-4201-a5e6-####/resourceGroups/*****-rg/providers/Microsoft.ContainerRegistry/registries/*****registry Error copying blobs. Error copying blobs. Error copying blobs. Error copying blobs. Error copying blobs.

Below image shows that I am able to download image directly with docker command.

Someone reported similar issue at https://github.com/Azure/acr/issues/632 as well but remains inconclusive.

Can someone please have a look why such a basic operation is failing? Also why the error message is so abstract for anyone to act on it?

Happy to share more details if needed.

4 answers

Your answer

Answer 1

Hi @AgBe DevOps

Thanks for raising your question to the Microsoft learn.

After making some analysis, this issue is because Quay does not honor requests for ranged blob queries, which ACR's import feature depends on. This is known issue with quay.io failing to support ranged queries.

For now, we recommend first pulling quay.io cert-manager images using docker, tagging them to ACR, and then pushing them to your ACR registry.

Using az acr command is going return the issue shared above by you and also from my side.

Using docker will mitigate the issue for you. Hope that helps.

Answer 2

Markus Blaschke (Daimler Truck AG) 1

@Eddie Neto Is there any ETA when this is fixed? This issue fills our ACRs with broken images and there is no good way to check which images are broken now.

Why is the ACR import relying on ranged blob queries? It's that an optional HTTP feature?

Pulling images using docker isn't a solution when dealing with multi-arch containers. This is a real blocker when dealing with quay images currently.

Eddie Neto 1,251 Reputation points Microsoft Employee

2023-01-10T19:25:32.53+00:00

Hi @Markus Blaschke (Daimler Truck AG)

Why is the ACR import relying on ranged blob queries? If we try to download the whole layer using one call the size of the layer, we can support will be limited by the memory size of the VM running our service. The way replication work is download 4MB of data and then write 4MB of data directly in storage hence it allows us to support very large layers. Also, the service is doing multiple downloads in parallel so we cannot deterministically identify how much memory will be available for thread to download the layer.

Is there any ETA when this is fixed? We´re currently talking with Quay to investigate this issue further. At this time, we do not have ETAs to share but will keep the thread updated with new information

Hope this helps. Please "Accept as Answer" if it helped, so that it can help others in the community looking for help on similar topics.
Markus Blaschke (Daimler Truck AG) 1 Reputation point

2023-01-11T08:31:22.7366667+00:00

@Eddie Neto

But that means ACRs are not fully HTTP compliant and more or less officially violating the RFCs 🤔

Why not use temp files for downloading big files? That's would be a better way then violating an official Internet standard and relying on optional HTTP features.

What about other registries which also don't support this? And it could break any time in future again so better not to rely on ACRs import feature again.

Are docker builds on ACRs also affected?
Eddie Neto 1,251 Reputation points Microsoft Employee

2023-01-12T16:26:26.8366667+00:00

Hi @Markus Blaschke (Daimler Truck AG)

Sorry for delay.

Why not use temp files for downloading big files?

We already use the temp file. This temp file is an append blob in the storage account and not on the service VM. This helps us support docker images having a large layer. If we store this temp file on the VM disk, the size of the docker layer supported by ACR will be restricted by the VM disk size.

We can use other options, like detecting if the server supports a range header. If not, we can pass on the flow to a different process to download the complete layer on call. We can set a memory limit for this process. We still need to explore these options.

Are docker builds on ACRs also affected?

It should not be affected.

What about other registries which also don't support this?

Right now, these registries will also face the same issue. We´re using this as an opportunity to improve the reliability of ACR import.

Answer 3

Markus Blaschke (Daimler Truck AG) 1

docker native workaround with multi arch container support (but requires "docker buildx"):

IMAGE_SRC=quay.io/foo/bar:v1.0 IMAGE_DEST=foo.azurecr.io/foo/bar:v1.0

docker buildx imagetools create --tag "$IMAGE_DEST" "$IMAGE_SRC"

Answer 4

Eddie Neto 1,251 Microsoft Employee

@AgBe DevOps / @Markus Blaschke (Daimler Truck AG)

Issue is mitigated for now. Just tested from my side and is working. Hope you can test from your side and let me know the result.

Hope this helps. Please "Accept as Answer" if it helped, so that it can help others in the community looking for help on similar topics.

Share via

Unable to import image to container registry

4 answers

Your answer