ACR Artifact Cache proxies images but never stores them — 0 tags/manifests on all cached repositories

Question

We have an Azure Container Registry (Premium SKU, UK South) with Artifact Cache configured using 5 cache rules (Docker Hub with credential set, GitHub, Quay, Kubernetes registry, Google Artifact Registry). All cache rules and the credential set show ProvisioningState: Succeeded.

Problem: Every image pull through the cache succeeds (images are delivered to the client), but nothing is persisted in ACR. All ~40 cached repositories show tagCount: 0 and manifestCount: 0. The repository namespace is created on first pull, but no manifests or tags are ever stored. This means every pull on a fresh node traverses the internet instead of being served from the local ACR private endpoint.

Diagnostic evidence:

• Enabled diagnostic settings with ContainerRegistryRepositoryEvents and ContainerRegistryLoginEvents categories
• ContainerRegistryLoginEvents are logged normally (containerd and docker clients authenticate successfully)
• ContainerRegistryRepositoryEvents container is completely empty — no Pull or Push events are generated for cache-proxied pulls
• This confirms ACR is not performing any store/write operations during cache pulls

What we've tried (no effect):

1. Disabled retention policy (retention_policy_in_days = 0) — suspected it was deleting untagged child manifests from multi-arch images (ref: Azure/acr-cli#131)
2. Deleted all stale 0-tag repository namespaces and recreated all cache rules via Terraform taint/apply (ref: Azure/acr#765, Azure/acr#789)
3. Tested with both containerd (kubelet on AKS) and Docker CLI pulls — same result
4. Verified AKS kubelet identity has AcrPull role on the ACR

Configuration:

• Premium SKU, public network access disabled, private endpoint enabled
• Anonymous pull enabled, RBAC-based auth (no admin user)
• Data endpoint enabled
• Docker Hub cache rule uses a credential set with Key Vault-stored credentials
• Other cache rules (ghcr.io, quay.io, registry.k8s.io, pkg.dev) have no credential set

Expected behavior: After a pull through a cache rule, az acr repository show should report manifestCount > 0 and tagCount > 0, and subsequent pulls should be served from local ACR storage.

Actual behavior: Repos are created with manifestCount: 0, tagCount: 0 indefinitely. The cache acts as a transparent proxy only.

Is this a known backend issue for the UK South region, or is there an additional configuration step required to enable actual caching?

Answer 1

Root Cause: networkRuleBypassOptions set to None

After extensive debugging, I found the root cause. The ACR had networkRuleBypassOptions set to None, which blocks Azure's own internal services — including the Artifact Cache service — from writing cached layers and manifests into the registry.

With this setting, the cache can proxy pulls (pass-through from upstream registries) but cannot store anything, because the cache service's write operations are blocked by the network rules. This explains the exact symptom: repositories get created (namespace creation works), but manifestCount and tagCount remain at 0, and no ContainerRegistryRepositoryEvents are ever generated.

How I confirmed it

1. Created a fresh ACR in West Europe with networkRuleBypassOptions: AzureServices — caching worked immediately (manifestCount: 1, tagCount: 1 after a pull).
2. Created another fresh ACR in UK South (same region as the broken one) with networkRuleBypassOptions:AzureServices — also worked. This ruled out a region-level issue.
3. Updated the existing broken ACR to set networkRuleBypassOptions: AzureServices — caching started working immediately.

The fix

az acr update --name <acr-name> --allow-trusted-services true

Or in Terraform/OpenTofu:

resource "azurerm_container_registry" "this" {
  # ...
  network_rule_bypass_option = "AzureServices"
}

Why this happened

If you're using the https://registry.terraform.io/modules/Azure/avm-res-containerregistry-registry/azurerm/latest (Azure/avm-res-containerregistry-registry/azurerm), the network_rule_bypass_option variable defaults to "None". Combined with public_network_access_enabled = false and network_rule_set.default_action = "Deny" (a typical private endpoint setup), this silently breaks Artifact Cache. The Azure documentation for Artifact Cache doesn't mention this requirement, which makes it easy to miss.