Container registry webhook to App Service with private endpoint gets 403 Forbidden

Question

Situation
We have: App Service with VNet integration and private endpoint, Application Gateway (for SSL offloading) routing public requests to App Service, DNS zone and Private DNS zone are set up. One picture for thousand words:

Problem
Continuous deployment process does not work: in the deployment center of the App Service I have configured to pull the image tagged as latest from the Container registry (it gets deployed just fine) and I have also checked the Continuous deployment "yes". The App Service created a webhook in Container registry's configuration.

• When a new image tagged as latest is pushed to Container registry the webhook is triggered but receives 403 Forbidden from the App Service's SCM endpoint.
• When trying to ping the webhook from Container registry's configuration page manually it also receives 403 Forbidden from the App Service's SCM endpoint.
• When making manual HTTP POST request from the Linux Virtual Machine (using wget or curl) it receives 401 Unauthorized from the App Service's SCM endpoint.

I have double checked:

• the username/password is correct
• the URL & Authorization header in wget/curl are correct
• application gateway with App Service works correctly (App Service's private endpoint is reachable)
• manual request (wget, curl) to App Service from Linux Virtual Machine also works just fine (App Service's private endpoint is reachable)
• DNS lookup for mywebapp.azurewebsites.net resolves correctly to private IP address
• DNS lookup for mywebapp.scm.azurewebsites.net resolves correctly to private IP address

Why we are always receiving 403 Forbidden / 401 Unauthorized from App Service's SCM endpoint?

Any help/ideas would be appreciated, I need to finish up setting the CD process.

Thank you.

Answer 1

Hi,

I have the same problem here with both WebApp and Container Registry having private endpoint.

I found 2 solutions

First One : Enable the WebApp Network Integration on a dedicated subnet and set a configuration value in Webapp

Webapp configuration Name : WEBSITE_PULL_IMAGE_OVER_VNET
Webapp configuration value : true

Second One : i considered this is a bug => so temporarly added the public IPs of the webapp to the ACR Enable Public access from selected networks

Waiting for MS to resolve without the need to create a subnet delegated zone for webapps.

Maybe there will be a conf variable like WEBSITE_PULL_IMAGE_OVER_PRIVATE_ENDPOINT :D

Answer 2

I have the same problem.

And to elaborate on this bug. I noticed that when the app is set as a private endpoint not only do continuous updates fail, in order to get deployment center for the app to pull an image manually I need to turn continuous deployment off. Then change the image. Then save. It will not pull with continuous deployment on.

Answer 3

Have the same problem.

App Service have private endpoint enabled. ACR tries to send webhook but gets HTTP 403.

On webhook details, on action details, it's possible to see that ACR is trying to connect to app service via public IP.

x-ms-forbidden-ip shows which IP app service is seeing.

Both acr and app service have private endpoints on (differente) spoke subscriptions, interconnected via hub.

Private DNS is at hub.

Answer 4

Same. Will try this tip:

Answer 5

I have the same problem.