Deploying multiple validating webhooks in the same AKS cluster

Question

I am working with an AKS cluster and AKS comes with an pre-deployed instance of Gatekeeper for validating webhooks.

Hence I am curious, how can one go about installing their own instance of Gatekeeper next to the one provided by AKS. Given that I am working on setting up a policy infrastructure for multi cloud (both on prem and cloud) using OPA and Gatekeeper, I wanted to keep the overall solution as much as cloud/platform agnostic, as possible. Additionally it also makes for a better developer experience where they can simply deploy their gatekeeper policies from CLI using kubectl instead of having to go through Azure Policy Engine.

Hence this got me thinking if I can deploy a separate instance of gatekeeper on the same cluster and create a new validating webhook configuration ? Would that even work ?

If yes, what all changes would need to be made.. Any thoughts ?

Answer 1

@Akshay Sinha , thank you for your question. Yes, multiple validating webhooks in the same AKS cluster is possible. Although you can install a second gatekeeper instance in a different namespace (the gatekeeper-controller-manager's spec.containers[name=manager].args should be appropriately set with --exempt-namespace) on the AKS cluster than the Azure Policy Addon's gatekeeper-system namespace, it would be completely unnecessary. The gatekeeper Deployments made by Azure Policy Addon is sufficient to accommodate your own OPA ConstraintTemplates and Constraints or Azure Custom policy definitions

If you have not enabled Azure Policy Addon for the AKS cluster, then you can follow these installation instructions for Gatekeeper on AKS

----

Some additional information:

What Kubernetes admission controllers does AKS support? Can admission controllers be added or removed?

AKS supports the following admission controllers:

• NamespaceLifecycle
• LimitRanger
• ServiceAccount
• DefaultStorageClass
• DefaultTolerationSeconds
• MutatingAdmissionWebhook
• ValidatingAdmissionWebhook
• ResourceQuota
• PodNodeSelector
• PodTolerationRestriction
• ExtendedResourceToleration

Currently, you can't modify the list of admission controllers in AKS.

Can I use admission controller webhooks on AKS?

Yes, you may use admission controller webhooks on AKS. It's recommended you exclude internal AKS namespaces, which are marked with the control-plane label. For example, by adding the below to the webhook configuration:

namespaceSelector:  
    matchExpressions:  
    - key: control-plane  
      operator: DoesNotExist  

AKS firewalls the API server egress so your admission controller webhooks need to be accessible from within the cluster.

Can admission controller webhooks impact kube-system and internal AKS namespaces?

To protect the stability of the system and prevent custom admission controllers from impacting internal services in the kube-system, namespace AKS has an Admissions Enforcer, which automatically excludes kube-system and AKS internal namespaces. This service ensures the custom admission controllers don't affect the services running in kube-system.

If you have a critical use case for having something deployed on kube-system (not recommended) which you require to be covered by your custom admission webhook, you may add the below label or annotation so that Admissions Enforcer ignores it.

Label: "admissions.enforcer/disabled": "true" or Annotation: "admissions.enforcer/disabled": true

----
Hope this helps.

Please "Accept as Answer" if it helped, so that it can help others in the community looking for help on similar topics.