How do I fix "Persistent Image Pull Errors for image" errors for my Container Apps app?

Question

How do I fix "Persistent Image Pull Errors for image" errors for my Container Apps app?

Brent Williams 20

Our staging and production Container Apps are unresponsive.

The System Log Stream for the production container app is displaying the following errors. I have not been able to resolve this issue on my own. Restarting the revision did not address the issue. The Logs do not display any application-related error.

Our staging server Container App is also down and displaying similar error messages it its log.

{"TimeStamp":"2025-04-07 22:26:13 \u002B0000 UTC","Type":"Normal","ContainerAppName":"aca-guardianfas-platform-prod","RevisionName":"aca-guardianfas-platform-prod--acg8fih","ReplicaName":"aca-guardianfas-platform-prod--acg8fih-5c4958b4cf-c8r4m","Msg":"Persistent Image Pull Errors for image \u0022100.100.0.217:443/registry/aca-guardianfas-platform-prod-buildd342e\u0022. Please check https://aka.ms/cappsimagepullerrors.","Reason":"ContainerBackOff","EventSource":"ContainerAppController","Count":372} {"TimeStamp":"2025-04-07 22:27:55 \u002B0000 UTC","Type":"Warning","ContainerAppName":"aca-guardianfas-platform-prod","RevisionName":"aca-guardianfas-platform-prod--acg8fih","ReplicaName":"aca-guardianfas-platform-prod--acg8fih-54d9947b85-ql249","Msg":"Failed to pull image \u0022100.100.0.217:443/registry/aca-guardianfas-platform-prod-buildd342e\u0022: rpc error: code = DeadlineExceeded desc = failed to pull and unpack image \u0022100.100.0.217:443/registry/aca-guardianfas-platform-prod-buildd342e:latest\u0022: failed to resolve reference \u0022100.100.0.217:443/registry/aca-guardianfas-platform-prod-buildd342e:latest\u0022: failed to do request: Head \u0022https://100.100.0.217:443/v2/registry/aca-guardianfas-platform-prod-buildd342e/manifests/latest\u0022: dial tcp 100.100.0.217:443: i/o timeout","Reason":"ContainerCrashing","EventSource":"ContainerAppController","Count":5} {"TimeStamp":"2025-04-07T22:28:56Z","Type":"Normal","ContainerAppName":null,"RevisionName":null,"ReplicaName":null,"Msg":"No events since last 60 seconds","Reason":"NoNewEvents","EventSource":"ContainerAppController","Count":1} {"TimeStamp":"2025-04-07T22:30:18Z","Type":"Normal","ContainerAppName":null,"RevisionName":null,"ReplicaName":null,"Msg":"No events since last 60 seconds","Reason":"NoNewEvents","EventSource":"ContainerAppController","Count":1} {"TimeStamp":"2025-04-07 22:31:14 \u002B0000 UTC","Type":"Normal","ContainerAppName":"aca-guardianfas-platform-prod","RevisionName":"aca-guardianfas-platform-prod--acg8fih","ReplicaName":"aca-guardianfas-platform-prod--acg8fih-5c4958b4cf-c8r4m","Msg":"Persistent Image Pull Errors for image \u0022100.100.0.217:443/registry/aca-guardianfas-platform-prod-buildd342e\u0022. Please check https://aka.ms/cappsimagepullerrors.","Reason":"ContainerBackOff","EventSource":"ContainerAppController","Count":392}

Loknathsatyasaivarma Mahali 2,740 Reputation points Microsoft External Staff Moderator

2025-04-08T00:22:22.6266667+00:00

Hello Brent Williams,

Thanks for reaching out, we are trying to find out the issue with the help of our internal tools, we request you to please allow us some time to thoroughly investigate the issue and provide you with the appropriate solution.
VenkateshDodda-MSFT 24,951 Reputation points Microsoft Employee Moderator

2025-04-08T05:43:44.7633333+00:00
@Brent Williams Thanks for your patience on this. Could you please follow the below troubleshooting guide and trouble shoot the issue by enabling diagnostic setting , followed by querying the data.

Troubleshoot using Image Pull Failures detector in Availability and performance under diagnose and solve problems blade of your container apps.

Troubleshoot Image pull errors in Azure container apps

Container Apps - Backoff restarts and Container exists.

let us know if you are still facing the issue.
Loknathsatyasaivarma Mahali 2,740 Reputation points Microsoft External Staff Moderator

2025-04-08T19:12:51.7833333+00:00

@Brent Williams, Just checking into see if the information provided the VenkateshDodda-MSFT, helps you in better understanding and solve your concerns, if you have any further concerns, please feel free to drop them here.
Brent Williams 20 Reputation points

2025-04-08T21:31:22.92+00:00
I have attempted to troubleshoot as suggested, but the "Availability and performance" request never completes analyzing the data or rendering the page.

However, I did look at the ContainerAppSystemLogs_CL in the Logs page. It appears that the issue is a networking issue, in which the image cannot be pulled from the registry.

I can see the first log entry in which a problem appears --

The "TimeGenerated [UTC]" and "time_t [UTC]" are 4/7/2025, 8:51:59.303 PM

This is consistent with our experience. Prior to this time, the application was available.

Here is the first error message:

Failed to pull image "100.100.0.217:443/registry/aca-guardianfas-platform-prod-buildd342e": rpc error: code = DeadlineExceeded desc = failed to pull and unpack image "100.100.0.217:443/registry/aca-guardianfas-platform-prod-buildd342e:latest": failed to resolve reference "100.100.0.217:443/registry/aca-guardianfas-platform-prod-buildd342e:latest": failed to do request: Head "https://100.100.0.217:443/v2/registry/aca-guardianfas-platform-prod-buildd342e/manifests/latest": dial tcp 100.100.0.217:443: i/o timeout

The troubleshooting web page which was shared w/ me suggests that this is a networking issue, like this example:

Failed to pull image "someacr.azurecr.io/someimage:sometag": rpc error: code = Canceled desc = failed to pull and unpack image "someacr.azurecr.io/someimage:sometag": failed to resolve reference "someacr.azurecr.io/someimage:sometag": failed to do request: Head "someacr.azurecr.io/someimage:sometag": context canceled

Please let me know if I can provide further info to you.

Thanks,

Brent Williams

Guardian FAS
Loknathsatyasaivarma Mahali 2,740 Reputation points Microsoft External Staff Moderator

2025-04-08T23:20:11.9966667+00:00

Hello Brent Williams,

Thanks for sharing the details, the error you're encountering (failed to pull image ...: i/o timeout) points to a networking issue when pulling the container image from the registry. This could be caused by network misconfigurations, such as blocked access through Network Security Group (NSG) rules or User Defined Routes (UDR), or firewall settings restricting access to the registry. It's essential to ensure the registry is reachable from your container’s network and check for any misconfigured NSG or UDR rules. Additionally, verify that DNS settings are correctly configured, and test the registry connection using curl or nslookup.

Another possibility is authentication failure, where the container cannot pull the image due to invalid credentials. Ensure that the appropriate credentials (username/password or Managed Identity with the AcrPull role) are in place, and that they are up to date. The ContainerAppSystemLogs_CL logs can provide more insights into the error, especially related to persistent image pull issues or ImagePullBackOff errors.

Finally, check the Azure Container Registry (ACR) configuration, ensuring that there are no firewall restrictions or private endpoint misconfigurations that might prevent access. If the issue continues, verify that ACR credentials are correctly configured and review the Azure status page for any registry outages.

Also check this Document which helps you in solving the issue.
Brent Williams 20 Reputation points

2025-04-09T02:19:45.6333333+00:00

Thank you for your response. I will review the network configuration of my production and staging ContainerApp apps and see if I can spot anything that correlates with the information you've shared in your comment or in the document.Please bear in mind that I did not make any changes to the configuration of my affected ContainerApps prior to this incident. In both instances, I had repeatedly and successfully uploaded and deployed new versions of my application to the ContainerApp environment, and the errors began appearing hours or days later after the deployments and my testing of them.

I do not, as a policy, make any changes to the configuration of my ContainerApp environments. Up to now, they have "just worked." All that I've needed to do for the last several months is to upload a new version of my application and periodically spot check the logs to ensure that everything is working as expected.

I suspect that a change was made to Azure's networking config, and that change is the root cause of the sudden failure of my two ContainerApps. I would not be surprised if other ContainerApps were reporting the same error.

The challenge that I face is that the ContainerApp product is a classic "PaaS" -- a platform-as-a-service -- offering, meaning that there are few opportunities for me to directly interact with the low-level configuration elements of the environment which may be the root cause of the issue.

Again, if I did not make any changes to any of the elements that you describe in your list of suggestions for me to check, why would my ContainerApps suddenly stop working?

I recommend that you check to ensure that some changes were not made to the networking config of my ContainerApps by Azure staff in the last 24 to 72 hours. There may be a more systemic error afoot affecting many ContainerApps of which Azure is not aware.

Thanks,

Brent Williams

Guardian FAS
Brent Williams 20 Reputation points

2025-04-09T02:44:46.2866667+00:00

I am going to take a guess here, based upon prior experience deploying container-based apps in AWS. When I look at the settings for Application > Containers > Properties, I see the following values:

Image source: Docker Hub or other registries

Image type: Public

Registry login server: 100.100.0.217:443

I suspect that a public container image is being used for my Container App, and that Azure is attempting to pull it from Docker Hub, and for whatever reason, that has started failing.

I suggest this because Docker Hub has, in the past, been difficult to work with through AWS, throttling traffic by IP address. I suggest you look into this.
Loknathsatyasaivarma Mahali 2,740 Reputation points Microsoft External Staff Moderator

2025-04-10T23:07:31.0766667+00:00

Hello Brent Williams,

Thanks for your patience on this and providing the details regarding the issue we are discussing this issue with our internal team and also, we have collaborated with the networking team to provide you with the effective solution.
Ryan Hill 30,281 Reputation points Microsoft Employee Moderator

2025-04-11T14:14:11.6366667+00:00

Hi @Brent Williams, I posted a private message to gather additional information from you.
Loknathsatyasaivarma Mahali 2,740 Reputation points Microsoft External Staff Moderator

2025-04-15T22:59:57.17+00:00

Hello Brent Williams,

We appreciate your patience. We have escalated this issue to our product group team, and discussions are ongoing. We will provide you with an update as soon as we receive from them.

Accepted answer

0 additional answers

Your answer

Loknathsatyasaivarma Mahali 2,740 Reputation points Microsoft External Staff Moderator

2025-04-08T00:22:22.6266667+00:00

Hello Brent Williams,

Thanks for reaching out, we are trying to find out the issue with the help of our internal tools, we request you to please allow us some time to thoroughly investigate the issue and provide you with the appropriate solution.
VenkateshDodda-MSFT 24,951 Reputation points Microsoft Employee Moderator

2025-04-08T05:43:44.7633333+00:00

@Brent Williams Thanks for your patience on this. Could you please follow the below troubleshooting guide and trouble shoot the issue by enabling diagnostic setting , followed by querying the data.

Troubleshoot using Image Pull Failures detector in Availability and performance under diagnose and solve problems blade of your container apps.

Troubleshoot Image pull errors in Azure container apps

Container Apps - Backoff restarts and Container exists.

let us know if you are still facing the issue.
Loknathsatyasaivarma Mahali 2,740 Reputation points Microsoft External Staff Moderator

2025-04-08T19:12:51.7833333+00:00

@Brent Williams, Just checking into see if the information provided the VenkateshDodda-MSFT, helps you in better understanding and solve your concerns, if you have any further concerns, please feel free to drop them here.
Brent Williams 20 Reputation points

2025-04-08T21:31:22.92+00:00

I have attempted to troubleshoot as suggested, but the "Availability and performance" request never completes analyzing the data or rendering the page.

However, I did look at the ContainerAppSystemLogs_CL in the Logs page. It appears that the issue is a networking issue, in which the image cannot be pulled from the registry.

I can see the first log entry in which a problem appears --

The "TimeGenerated [UTC]" and "time_t [UTC]" are 4/7/2025, 8:51:59.303 PM

This is consistent with our experience. Prior to this time, the application was available.

Here is the first error message:

Failed to pull image "100.100.0.217:443/registry/aca-guardianfas-platform-prod-buildd342e": rpc error: code = DeadlineExceeded desc = failed to pull and unpack image "100.100.0.217:443/registry/aca-guardianfas-platform-prod-buildd342e:latest": failed to resolve reference "100.100.0.217:443/registry/aca-guardianfas-platform-prod-buildd342e:latest": failed to do request: Head "https://100.100.0.217:443/v2/registry/aca-guardianfas-platform-prod-buildd342e/manifests/latest": dial tcp 100.100.0.217:443: i/o timeout

The troubleshooting web page which was shared w/ me suggests that this is a networking issue, like this example:

Failed to pull image "someacr.azurecr.io/someimage:sometag": rpc error: code = Canceled desc = failed to pull and unpack image "someacr.azurecr.io/someimage:sometag": failed to resolve reference "someacr.azurecr.io/someimage:sometag": failed to do request: Head "someacr.azurecr.io/someimage:sometag": context canceled

Please let me know if I can provide further info to you.

Thanks,

Brent Williams

Guardian FAS
Loknathsatyasaivarma Mahali 2,740 Reputation points Microsoft External Staff Moderator

2025-04-08T23:20:11.9966667+00:00

Hello Brent Williams,

Thanks for sharing the details, the error you're encountering (failed to pull image ...: i/o timeout) points to a networking issue when pulling the container image from the registry. This could be caused by network misconfigurations, such as blocked access through Network Security Group (NSG) rules or User Defined Routes (UDR), or firewall settings restricting access to the registry. It's essential to ensure the registry is reachable from your container’s network and check for any misconfigured NSG or UDR rules. Additionally, verify that DNS settings are correctly configured, and test the registry connection using curl or nslookup.

Another possibility is authentication failure, where the container cannot pull the image due to invalid credentials. Ensure that the appropriate credentials (username/password or Managed Identity with the AcrPull role) are in place, and that they are up to date. The ContainerAppSystemLogs_CL logs can provide more insights into the error, especially related to persistent image pull issues or ImagePullBackOff errors.

Finally, check the Azure Container Registry (ACR) configuration, ensuring that there are no firewall restrictions or private endpoint misconfigurations that might prevent access. If the issue continues, verify that ACR credentials are correctly configured and review the Azure status page for any registry outages.

Also check this Document which helps you in solving the issue.
Brent Williams 20 Reputation points

2025-04-09T02:19:45.6333333+00:00

Thank you for your response. I will review the network configuration of my production and staging ContainerApp apps and see if I can spot anything that correlates with the information you've shared in your comment or in the document.Please bear in mind that I did not make any changes to the configuration of my affected ContainerApps prior to this incident. In both instances, I had repeatedly and successfully uploaded and deployed new versions of my application to the ContainerApp environment, and the errors began appearing hours or days later after the deployments and my testing of them.

I do not, as a policy, make any changes to the configuration of my ContainerApp environments. Up to now, they have "just worked." All that I've needed to do for the last several months is to upload a new version of my application and periodically spot check the logs to ensure that everything is working as expected.

I suspect that a change was made to Azure's networking config, and that change is the root cause of the sudden failure of my two ContainerApps. I would not be surprised if other ContainerApps were reporting the same error.

The challenge that I face is that the ContainerApp product is a classic "PaaS" -- a platform-as-a-service -- offering, meaning that there are few opportunities for me to directly interact with the low-level configuration elements of the environment which may be the root cause of the issue.

Again, if I did not make any changes to any of the elements that you describe in your list of suggestions for me to check, why would my ContainerApps suddenly stop working?

I recommend that you check to ensure that some changes were not made to the networking config of my ContainerApps by Azure staff in the last 24 to 72 hours. There may be a more systemic error afoot affecting many ContainerApps of which Azure is not aware.

Thanks,

Brent Williams

Guardian FAS
Brent Williams 20 Reputation points

2025-04-09T02:44:46.2866667+00:00

I am going to take a guess here, based upon prior experience deploying container-based apps in AWS. When I look at the settings for Application > Containers > Properties, I see the following values:

Image source: Docker Hub or other registries

Image type: Public

Registry login server: 100.100.0.217:443

I suspect that a public container image is being used for my Container App, and that Azure is attempting to pull it from Docker Hub, and for whatever reason, that has started failing.

I suggest this because Docker Hub has, in the past, been difficult to work with through AWS, throttling traffic by IP address. I suggest you look into this.
Loknathsatyasaivarma Mahali 2,740 Reputation points Microsoft External Staff Moderator

2025-04-10T23:07:31.0766667+00:00

Hello Brent Williams,

Thanks for your patience on this and providing the details regarding the issue we are discussing this issue with our internal team and also, we have collaborated with the networking team to provide you with the effective solution.
Ryan Hill 30,281 Reputation points Microsoft Employee Moderator

2025-04-11T14:14:11.6366667+00:00

Hi @Brent Williams, I posted a private message to gather additional information from you.
Loknathsatyasaivarma Mahali 2,740 Reputation points Microsoft External Staff Moderator

2025-04-15T22:59:57.17+00:00

Hello Brent Williams,

We appreciate your patience. We have escalated this issue to our product group team, and discussions are ongoing. We will provide you with an update as soon as we receive from them.

Answer 1

Hi @Brent Williams

I want to thank you for your patience while we worked offline to determine the issue. According to the product group, the Artifact Deployment is a preview feature that has a known issue with VNET integration. With the feature being in preview, the best path forward to get unblocked is to configure your application to pull from a built image.

You were able to successfully to put your app in a Dockerfile, build an image that satisfied your Spring Boot app, and updated your container app to leverage your built image.

Share via

How do I fix "Persistent Image Pull Errors for image" errors for my Container Apps app?

0 additional answers

Your answer