Why are all ACR push and pulls receiving 500 Internal Server Error?

Question

All push and pull requests are receiving:

Error response from daemon: received unexpected HTTP status: 500 Internal Server Error

Started Friday, Feb 20th around 10PM EST. No changes were made to the ACR or its nearby components. The same error occurs for humans and managed identites.

We are able to browse repositories in the UI, but selecting a specific image reproduces the 500 Internal Server Error. The Artifact Manifest renders as blank, and the Manifest is also blank. The other metadata loads correctly. This happens for all images in all repositories.

The 500 errors are not visible in the Azure Diagnostics.

Answer 1

Hello @Jesse Silverstein ,

It seems like you're running into some frustrating issues with your Azure Container Registry (ACR) receiving 500 Internal Server Errors for push and pull requests. This error can be a bit tricky, but here are some potential steps to troubleshoot the problem:

Health Check: You mentioned that az acr health-check succeeds but with a warning regarding the notary. While the notary issue might be a red herring, it’s important to ensure that the ACR itself is healthy. You can run the command:

az acr check-health --name <yourRegistry> --ignore-errors --yes

Check if any errors related to this or others show up in the output.

Authentication Validation: Double-check the integrity of your authentication setup for ACR. Make sure that your credentials are valid and haven't expired. You can refer to the guide on troubleshooting login issues for more insights.

Repository Status: It's also crucial to check that none of your repositories or images are locked. You can do this by running:

az acr repository show --name <yourRegistry> --repository <yourRepository> --output jsonc

If the writeEnabled attribute is set to false, the repository is locked; you can unlock it with:

az acr repository update --name <yourRegistry> --repository <yourRepository> --write-enabled true

Storage Limits: Verify that your container registry hasn't reached its storage limit of 40 TiB. If it has, you won't be able to push additional data. You can look at your current storage usage to see if this could be a factor.

Network Connectivity: Since the issue arose without any clear changes being made, ensure that there aren't any underlying network connectivity issues. You can test your connection to the registry by running:

az acr check-health -n <yourRegistry>

If none of these suggestions work, you might want to gather additional information like logs or check if there are any ongoing service interruptions impacting ACR in your region.

I hope this helps you troubleshoot the problem effectively! If you need more assistance, feel free to ask.

Answer 2

The 500 Internal Server Error you are experiencing with Azure Container Registry (ACR) push and pull requests can be attributed to several potential issues. Here are some steps you can take to troubleshoot the problem:

1. Check Container Registry Health: Start by checking the health of your container registry. You can run the following command to verify its status:

      az acr check-health --name <myregistry> --ignore-errors --yes
   

   This command will provide you with any error codes and descriptions that may help identify the issue.
2. Inspect for Storage Limit Issues: Ensure that your container registry has not reached its storage limit, which is capped at 40 TiB. If the storage limit is reached, it could prevent push and pull operations.
3. Authentication Issues: Verify that your authentication is correctly set up. If there are issues with authentication, it could lead to errors when attempting to push or pull images. Refer to the Azure documentation on Authenticating with an Azure container registry for guidance.
4. Review Azure Diagnostics: Since you mentioned that the 500 errors are not visible in Azure Diagnostics, ensure that logging is properly configured to capture these errors. This might provide more insight into the underlying cause.
5. Retry the Operations: Sometimes, transient errors can cause 500 Internal Server Errors. Retrying the push or pull operations after a short period may resolve the issue.

If the problem persists after these checks, consider reaching out to Azure support for further assistance, as they may have more detailed insights into the service's current status and any ongoing issues.