question

specialsnowflake-6537 avatar image
0 Votes"
specialsnowflake-6537 asked objectclass commented

App Service for Web App for Containers, Container Registry and Private Endpoints

I have created in our Azure subscription an App Service (Web App for Containers, single container, App Service plan is P1v2) and a Container Registry (Premium). I have connected both the App Service and the ACR to a VNet using private endpoints. I have also configured VNet Integration for the App Service to the VNet.

When I set the Public Access of the ACR to disabled, I expect to force the App Service to pull its image from the ACR using only the networking of the VNet. Instead I get an error when trying to pull the image:

ERROR - DockerApiException: Docker API responded with status code=InternalServerError, response={"message":"Get https://my-acr.azurecr.io/v2/my-acr-repo/manifests/latest: denied: client with IP 'XX.XXX.XX.XXX' is not allowed access. Refer https://aka.ms/acr/firewall to grant access."}

If I then set the Public Access of the ACR to "Selected networks" and allow the IP address listed in the error above, it works.

My questions:

  1. Is the private endpoint scenario above not supported (App Service for Containers accessing ACR)? If not, is it being worked on for support and when will it be supported?

  2. If the above is not supported, how do I use Azure CLI to get the above IP address besides waiting for an error and pulling it from the text of the error? It's not the IP address associated with any of the virtual NICs nor the IP Address that nslookup resolves to when looking up the public host listed in the URL for the App Service.

Thanks for your help.

azure-webappsazure-container-registryazure-webapps-vnet
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

So we are getting closer to June, are you getting any closer to allowing vnet integrated app service plans to pull containers from private link enabled container registry ? Is there some where we track the progress on this, such as github?

2 Votes 2 ·
prmanhas-MSFT avatar image
2 Votes"
prmanhas-MSFT answered prmanhas-MSFT commented

@specialsnowflake-6537 I had discussion internally and below is the response I got:

As noted earlier – the private link scenario isn’t supported yet on App Service. And beyond we’re working on it and hope to land it sometime between now and end of June, there is not a more specific ETA.

What the customer might be seeing is one of the pool of outbound IP addresses used the App Service scale unit where the app is running. You can adding all of the outbound IP addresses associated with the app to their address allow list in ACR.

When looking at your outbound addresses you will see that there are two sets. If you look in your app Properties you can see them or use the command line items referred above.


When you set up a firewall though, use the Additional Outbound IP Addresses or possibleOutboundAddresses, however it shows up. It is the superset of what is possible for your app to use. That way if you scale it up or down across SKUs, it will still work.


Hope it helps!!!

Please "Accept as Answer" if it helped so it can help others in community looking for help on similar topics




· 4
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@prmanhas-MSFT - Any update on the progress of this issue? You said the end of June so I'm following up to see if support for this scenario has been added.

1 Vote 1 ·

Thanks @prmanhas-MSFT . This is the answer I was looking for on getting the list of possible outbound IP addresses even if it wasn't the answer I wanted around private end points w/ App Service for Containers.

0 Votes 0 ·

Hi,
is there any update on this topic? apparently even service endpoint is not working so we have to whitelist long list of IPs (and moreover it is not even possible to add a comment).

Thank you!
Fabio

0 Votes 0 ·

@PerfettiFabio-2926 I am reaching out to internal team to get updates on the issue.

Thanks

0 Votes 0 ·
camargoreislucas avatar image
1 Vote"
camargoreislucas answered specialsnowflake-6537 commented

Hi @specialsnowflake-6537

Can you confirm if you use a Azure DNS Zones resolving IPs for yours Private Links inside a VNET?

To access a resource in Azure using a Private Link you need to integrate with a Azure DNS Zones or configure your DNS Servers like this documentation:
https://docs.microsoft.com/en-us/azure/private-link/private-endpoint-dns#dns-configuration-scenarios

· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Yes - we have set up Azure DNS zones for the ACR (privatelink.azurecr.io) where we set up an A record for the ACR (my-acr in the above example) to point to the IP address of the private endpoint assigned to the ACR. Note we verified that we can pull from the ACR when public access is disabled using the above private link from a VM in the same subnet. It's the App Service that can't access it. I also can use the App Service to access other services via private link (Key Vault and our DB). It's just the combination of App Service for Containers, Azure Container Registry and Private Endpoints that doesn't work.

0 Votes 0 ·

I accidentally upvoted the above answer. Just to be clear, adding a DNS zone for the ACR has already been done and is not a solution to the stated problem.

0 Votes 0 ·
prmanhas-MSFT avatar image
0 Votes"
prmanhas-MSFT answered SureshBettadapur-4155 commented

@specialsnowflake-6537 Apologies for the delay in response and all the inconvenience caused because of the issue.

I had discussion internally and got to know It is not supported yet. This indeed is on roadmap and our engineering team is working on same but currently I dont have an ETA to share.

Hope it helps!!!

Please "Accept as Answer" if it helped so it can help others in community looking for help on similar topics

· 4
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@prmanhas-MSFT Thanks for responding with a definitive answer.

Do you have any idea how I can at least get the IP address that the App Service will use when requesting to pull an image from the specified ACR via Azure CLI? Currently I have to wait for the pull request to fail and look up the IP address in the App Service docker log file. We have found this IP address also changes over time. I tried adding a NAT Gateway with a static IP address assigned to see if the pull request would come from that IP address. Unfortunately it seems the pull request by the App Service doesn't go out via the NAT Gateway.

0 Votes 0 ·

Hi

How about App service/ Function app on ASE?

We have this scenario and when we disable public access to ACR, we still see the requests going with outbound public IP of the ASE. When I temporarily add the public IP in the list of allowed IPs on ACR, it goes through. But, I think this is not correct. ACR is on Private endpoint and it's in same VNET as ASE

0 Votes 0 ·

@prmanhas-MSFT can you kindly comment?

0 Votes 0 ·

Thanks for the response ...... as I understand, we need to include outbound public IP in the allowed list on the ACR .......

0 Votes 0 ·
AbhilashKonnur-4971 avatar image
0 Votes"
AbhilashKonnur-4971 answered

@specialsnowflake-6537 not sure if you already found the answer. The outbound IP's of App Service can be found in Properties blade.
95582-image.png



image.png (20.5 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

KiZac avatar image
1 Vote"
KiZac answered objectclass commented

See this blog for solution:

https://azure.github.io/AppService/2021/07/03/Linux-container-from-ACR-with-private-endpoint.html

The WEBSITE_PULL_IMAGE_OVER_VNET setting is what you need.

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Confirmed, many thanks.

0 Votes 0 ·