Azure AKS policies prohibit deployment of custom gatekeeper policies using kubectl

Akshay Sinha 56 Reputation points
2021-09-27T09:54:06.257+00:00

I am working with an AKS cluster and AKS comes with an pre-deployed instance of Gatekeeper for validating webhooks, in case you have policy Add-on enabled.

Hence I am curious, how can one go about installing their own instance of Gatekeeper next to the one provided by AKS. Given that I am working on setting up a policy infrastructure for multi cloud (both on prem and cloud) using OPA and Gatekeeper, I wanted to keep the overall solution as much as cloud/platform agnostic, as possible. Additionally it also makes for a better developer experience where they can simply deploy their gatekeeper policies from CLI using kubectl instead of having to go through Azure Policy Engine.

Hence this got me thinking if I can deploy a separate instance of gatekeeper on the same cluster and create a new validating webhook configuration.

The reason why I wanted to deploy a second instance of gatekeeper was to keep the developer experience intact, where they can still deploy their policies to a K8s cluster using kubectl like they do with other K8s systems currently. Additionally this also allows us to keep the overall policy solution as much as cloud/platform agnostic, as possible. We operate a multi cloud/on-prem infrastructure for K8s.

So I went ahead and deployed a new instance of Gatekeeper (after updating the existing webhook with --exempt-namespace arg). However when trying to apply a "constraint template" , I still get following error : 

admission webhook "byovalidation.policy.azure.com" denied the request: This cluster is governed by Azure Policy. Policies must be created through Azure.

The above error is coming from custom azure webhook azure-policy-validating-webhook-configuration. It seems likely that this webhook rejects any policies that are not coming in through Azure policy portal. Is that correct ?

According to K8s doc, it says that one can definitely have more than one validating webhooks deployed in the same cluster, but for a request to be allowed, all validating webhooks would need to reply with either "allow" or "I don't know". However, In our case, since one webhook would always reject the request, the whole request would always get rejected.

I understand that disabling the policy add-on for AKS as a whole would allow us to achieve what we are looking for, but we would like to avoid that option if possible.

Hence what might be the best way forward. Thoughts ?

PS : @SRIJIT-BOSE-MSFT This question is an follow up to an earlier question, https://learn.microsoft.com/en-us/answers/questions/563748/deploying-multiple-validating-webhooks-in-the-same.html. For Some strange reason, site wouldn't allow me to add any comments on the answer received. Hence adding a new question here with additional details.

Azure Kubernetes Service (AKS)
Azure Kubernetes Service (AKS)
An Azure service that provides serverless Kubernetes, an integrated continuous integration and continuous delivery experience, and enterprise-grade security and governance.
1,852 questions
Azure Policy
Azure Policy
An Azure service that is used to implement corporate governance and standards at scale for Azure resources.
790 questions
0 comments
Accepted answer
  1. SRIJIT-BOSE-MSFT 4,326 Reputation points Microsoft Employee
    2021-09-27T11:14:01.633+00:00

    @Akshay Sinha , thank you for sharing your concern here. We regret the issues you faced.

    We followed up internally on https://learn.microsoft.com/en-us/answers/questions/563748/deploying-multiple-validating-webhooks-in-the-same.html

    According to the updated limitations of Azure Policy Add-on for Kubernetes clusters, at the time of writing:

    Installations of Gatekeeper outside of the Azure Policy Add-on aren't supported. Uninstall any components installed by a previous Gatekeeper installation before enabling the Azure Policy Add-on.

    For now, as you mentioned, disabling the Azure policy add-on for AKS as a whole would allow you to bring in your Gatekeeper and deploy your own Constraint Template (which does not have to be through Azure).

    ----
    Hope this helps.

    Please "Accept as Answer" if it helped, so that it can help others in the community looking for help on similar topics.

    0 comments

1 additional answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. Miqdaad Patwa 1 Reputation point
    2022-07-20T07:24:24.587+00:00

    Hello @Akshay Sinha @SRIJIT-BOSE-MSFT ,

    I am also facing the same issue. We have created the custom policy in Azure and deployed the constraint templates. Now we are trying to edit the constraint template we are facing this error.

    admission webhook "byovalidation.policy.azure.com" denied the request: This cluster is governed by Azure Policy. Policies must be created through Azure.

    The above error is coming from custom azure webhook azure-policy-validating-webhook-configuration. It seems likely that this webhook rejects any policies that are not coming in through Azure policy portal.

    Also there are deployments and replica set which gets created when azure policy add on is enabled.

    kube-system pod/azure-policy-795c78444-phjl2 1/1 Running 0 8d
    kube-system pod/azure-policy-webhook-84884d989b-zmqbd 1/1 Running 0 16h
    kube-system service/azure-policy-webhook-service ClusterIP 192.168.66.219 <none> 443/TCP 16h
    kube-system deployment.apps/azure-policy 1/1 1 1 8d
    kube-system deployment.apps/azure-policy-webhook 1/1 1 1 16h
    kube-system replicaset.apps/azure-policy-795c78444 1 1 1 8d
    kube-system replicaset.apps/azure-policy-webhook-84884d989b 1 1 1 16h

    We tried deleting the webhooks but it reappeared again. Can you please help on the same if we can fix this error.

    If the existing webhook can be modified or completely removed.

    Note: We want the Azure Add on to be enabled as part of the security requirement.

    Please help

    0 comments