Dynamic Admission Webhook fails to post to external URL

Rahul Rai 1 Reputation point MVP
2020-08-18T01:42:00.827+00:00

Timeout on POST request to external URL:

I am following this guide to add a Validatingwebhookconfiguration to my AKS cluster. Here is my configuration which works on my local Kubernetes setup with Kind or Docker-desktop.

apiVersion: admissionregistration.k8s.io/v1beta1
kind: ValidatingWebhookConfiguration
metadata:
  name: compliance
webhooks:
  - name: compliance.custom.azure.com
    clientConfig:
      url: https://azure-fx.azurewebsites.net/api/AdmissionControlFx 
    rules:
      - apiGroups: ["apps"]
        apiVersions: ["*"]
        operations: ["CREATE","UPDATE"]
        resources: ["deployments"]
        scope: "Namespaced"
    timeoutSeconds: 30
    failurePolicy: Fail
    sideEffects: None
    admissionReviewVersions: ["v1", "v1beta1"]

On AKS, the api server is able to resolve the IP address of the host but times out on sending a POST request to the URL mentioned. I suspect there is some firewall policy that is not letting api server communicate with external urls.

What you expected to happen:

I expect the validatingwebhookconfiguration to send a POST request to the URL mentioned.

How to reproduce it (as minimally and precisely as possible):

Apply the configuration to AKS after changing the URL to any service e.g. Request bin. You will receive a timeout error when you try to kick off a deployment. The error message contains the correct IP address of the target service, which means that DNS resolution works fine. Just the POST request to endpoint fails.

Anything else we need to know?:
Here are the logs from the API server which show timeouts.

90349916-cf801400-e07e-11ea-81b0-c5e11723cc42.png

To rule out issues with not setting the caBundle property, I base 64 encoded the CA certificate *.azurewebsites.net as well, which still did not work.

Environment:

Kubernetes version (use kubectl version): 1.17.9
Size of cluster (how many worker nodes are in the cluster?): 1
General description of workloads in the cluster (e.g. HTTP microservices, Java app, Ruby on Rails, machine learning, etc.): NA
Others: NA

Azure Kubernetes Service (AKS)
Azure Kubernetes Service (AKS)
An Azure service that provides serverless Kubernetes, an integrated continuous integration and continuous delivery experience, and enterprise-grade security and governance.
2,146 questions
{count} votes

Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.