Troubleshooting Slow Container Pull Times and Failures in Azure and Local Networks

Anonymous
2023-04-19T16:33:11.0033333+00:00

We have been using Azure containers to pull an image that is around 11GB in size for some time now, and it always takes around 10 to 11 minutes to pull the container. However, locally, on a local network using a container registry on a Docker from one machine to another, pulling the container from scratch takes less than 3 minutes. Now, the other problem that we are facing is that the container pull is failing several times, and we can see this in the container description. This causes delays, and in one case, it took us 39 minutes to start the pod. What can be done to reduce the time it takes to pull the container? What transfer speed does the Microsoft container registry have that takes 11 minutes to pull the container? User's image

Azure Container Registry
Azure Container Registry
An Azure service that provides a registry of Docker and Open Container Initiative images.
406 questions
0 comments No comments
{count} votes

1 answer

Sort by: Most helpful
  1. deherman-MSFT 34,026 Reputation points Microsoft Employee
    2023-04-24T16:27:53.5866667+00:00

    @Anonymous

    You can find troubleshooting steps on the Troubleshoot registry performance and Troubleshoot common issues in Azure Container Instances pages. To improve container pull times and reduce failures, consider the following suggestions:

    1. Optimize the container image size: Large container images take longer to pull. Use multi-stage builds to ensure that the final image contains only the artifacts needed for your application, and not any extra content required at build time.
    2. Host the container image in the same region: Host the container image in Azure Container Registry in the same region where you intend to deploy container instances. This shortens the network path that the container image needs to travel, significantly shortening the download time.
    3. Leverage cached images: Azure Container Instances uses a caching mechanism to help speed container startup time for images built on common base images. For an up-to-date list of cached images and tags, use the List Cached Images API.
    4. Check network throttling: Check with your networking provider about network throttling that may occur with many concurrent operations. Review your Docker daemon configuration for the maximum concurrent uploads or downloads for each push or pull operation on the client. Configure higher limits if needed.
    5. Monitor and troubleshoot: Monitor the registry's StorageUsed metric for up-to-date data on storage consumption. If you experience issues with throughput to your registry, see the Troubleshoot registry performance guide.

    By optimizing the container image size, hosting the image in the same region, leveraging cached images, checking network throttling, and monitoring and troubleshooting, you can potentially reduce the time it takes to pull the container and minimize failures.

    If you are still facing issues and seeing failures I recommend creating an Azure support ticket so we can investigate in more detail. If you don't have a support plan please let me know and I can assist you in getting one created.


    Please remember to "Accept Answer" if any answer/reply helped, so that others in the community facing similar issues can easily find the solution.

    0 comments No comments