Share via

Container registry webhook to App Service with private endpoint gets 403 Forbidden

Marek Čambál 11 Reputation points
2021-03-18T12:41:55.217+00:00

Situation
We have: App Service with VNet integration and private endpoint, Application Gateway (for SSL offloading) routing public requests to App Service, DNS zone and Private DNS zone are set up. One picture for thousand words:

79169-webhook-problem.png

Problem
Continuous deployment process does not work: in the deployment center of the App Service I have configured to pull the image tagged as latest from the Container registry (it gets deployed just fine) and I have also checked the Continuous deployment "yes". The App Service created a webhook in Container registry's configuration.

  • When a new image tagged as latest is pushed to Container registry the webhook is triggered but receives 403 Forbidden from the App Service's SCM endpoint.
  • When trying to ping the webhook from Container registry's configuration page manually it also receives 403 Forbidden from the App Service's SCM endpoint.
  • When making manual HTTP POST request from the Linux Virtual Machine (using wget or curl) it receives 401 Unauthorized from the App Service's SCM endpoint.

I have double checked:

  • the username/password is correct
  • the URL & Authorization header in wget/curl are correct
  • application gateway with App Service works correctly (App Service's private endpoint is reachable)
  • manual request (wget, curl) to App Service from Linux Virtual Machine also works just fine (App Service's private endpoint is reachable)
  • DNS lookup for mywebapp.azurewebsites.net resolves correctly to private IP address
  • DNS lookup for mywebapp.scm.azurewebsites.net resolves correctly to private IP address

Why we are always receiving 403 Forbidden / 401 Unauthorized from App Service's SCM endpoint?

Any help/ideas would be appreciated, I need to finish up setting the CD process.

Thank you.

Azure Container Registry
Azure Container Registry

An Azure service that provides a registry of Docker and Open Container Initiative images.

Azure Private Link
Azure Private Link

An Azure service that provides private connectivity from a virtual network to Azure platform as a service, customer-owned, or Microsoft partner services.

Azure App Service
Azure App Service

Azure App Service is a service used to create and deploy scalable, mission-critical web apps.

6 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Sani Elfishawy 11 Reputation points
    2022-06-09T20:48:47.357+00:00

    I have the same problem.

    And to elaborate on this bug. I noticed that when the app is set as a private endpoint not only do continuous updates fail, in order to get deployment center for the app to pull an image manually I need to turn continuous deployment off. Then change the image. Then save. It will not pull with continuous deployment on.

    1 person found this answer helpful.
    0 comments
  2. Ivan PATILLON 6 Reputation points
    2022-05-18T08:53:46.66+00:00

    Hi,

    I have the same problem here with both WebApp and Container Registry having private endpoint.

    I found 2 solutions

    First One : Enable the WebApp Network Integration on a dedicated subnet and set a configuration value in Webapp

    Webapp configuration Name : WEBSITE_PULL_IMAGE_OVER_VNET
    Webapp configuration value : true

    Second One : i considered this is a bug => so temporarly added the public IPs of the webapp to the ACR Enable Public access from selected networks

    Waiting for MS to resolve without the need to create a subnet delegated zone for webapps.

    Maybe there will be a conf variable like WEBSITE_PULL_IMAGE_OVER_PRIVATE_ENDPOINT :D

    1 person found this answer helpful.
  3. Welton Rodrigo Torres Nascimento 6 Reputation points
    2022-02-15T23:31:07.613+00:00

    Have the same problem.

    App Service have private endpoint enabled. ACR tries to send webhook but gets HTTP 403.

    On webhook details, on action details, it's possible to see that ACR is trying to connect to app service via public IP.

    x-ms-forbidden-ip shows which IP app service is seeing.

    Both acr and app service have private endpoints on (differente) spoke subscriptions, interconnected via hub.

    Private DNS is at hub.

    174681-captura-de-tela-2022-02-15-as-202943.png

    1 person found this answer helpful.
  4. Sani Elfishawy 11 Reputation points
    2022-07-06T01:50:48.473+00:00

    This unfortunate behavior requiring ACR web-hook endpoints to be publicly available is documented as Microsoft support pointed out to me in this message:

    ------

    Hello Team,

    I hope your day is going well! We have an update for you! As of right now this isnt possible from an ACR perspective. "The endpoint for a webhook must be publicly accessible from the registry." Per Webhooks to respond to registry actions - Azure Container Registry | Microsoft Learn ( https://learn.microsoft.com/en-us/azure/container-registry/container-registry-webhook ) Right now I am trying to check if this will be possible in the future and if so when. I don't see it on the current road map but I want to double check. Here is the roadmap if you are curious acr/acr-roadmap.md at main · Azure/acr · GitHub. Once I have an update on this I will reach out to you immediately.

    Thank you,
    Jacob Blasingame

  5. Richard 1 Reputation point
    2022-04-30T08:32:01.8+00:00

    Same. Will try this tip:
    197915-image.png

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.