Configure group Managed Service Accounts (gMSA) for Windows containers with Azure Kubernetes Service on Azure Stack HCI and Windows Server
Applies to: Azure Stack HCI and Windows Server
To use AD Authentication, you can configure group Managed Service Accounts (gMSA) for Windows containers to run with a non-domain joined host. A group Managed Service Account is a special type of service account introduced in Windows Server 2012 that's designed to allow multiple computers to share an identity without knowing the password. Windows containers cannot be domain joined, but many Windows applications that run in Windows containers still need AD Authentication.
To learn how the Kubernetes community supports using gMSA with Windows containers, see Configure gMSA.
Architecture of gMSA for containers with a non-domain joined host
gMSA for containers with a non-domain joined host uses a portable user identity instead of a host identity to retrieve gMSA credentials. Therefore, manually joining Windows worker nodes to a domain is no longer necessary. The user identity is saved as a secret in Kubernetes. The diagram below shows the process for configuring gMSA for containers with a non-domain joined host:
gMSA for containers with a non-domain joined host provides the flexibility of creating containers with gMSA without joining the host node to the domain. Starting in Windows Server 2019, ccg.exe is supported, which enables a plug-in mechanism to retrieve gMSA credentials from Active Directory. You can use that identity to start the container. For more information on ccg.exe, see ICcgDomainAuthCredentials interface.
Comparison of gMSA for containers with a non-domain joined host and a domain joined host
When gMSA was initially introduced, it required the container host to be domain joined, which created a lot of overhead to join Windows worker nodes manually to a domain. This limitation has been addressed with gMSA for containers with a non-domain joined host, so users can now use gMSA with domain-unjoined hosts. Other improvements to gMSA include the following:
- Eliminating the requirement to manually join Windows worker nodes to a domain, which caused a lot of overhead. For scaling scenarios, this will simplify the process.
- In rolling update scenarios, users no longer need to rejoin the node to a domain.
- An easier process for managing the worker node machine accounts to retrieve gMSA service account passwords.
- A less complicated end-to-end process to configure gMSA with Kubernetes.
Before you begin
To run a Windows container with a group managed service account, you need the following:
- An Active Directory domain with at least one domain controller running Windows Server 2012 or later. There are no forest or domain functional level requirements to use gMSAs, but gMSA passwords can only be distributed by domain controllers running Windows Server 2012 or later. For more information, see Active Directory requirements for gMSAs.
- Permission to create a gMSA account. To create a gMSA account, you'll need to be a Domain Administrator or use an account that has been given the permission to create msDS-GroupManagedServiceAccount objects.
- Access to the internet to download the CredentialSpec PowerShell module. If you're working in a disconnected environment, you can save the module on a computer with internet access and copy it to your development machine or container host.
- To ensure gMSA and AD authentication work, ensure that the Azure Stack HCI and Windows Server cluster nodes are configured to synchronize their time with either a domain controller or other time source. You should also make sure Hyper-V is configured to synchronize time to any virtual machines.
Prepare the gMSA in the domain controller
Follow the steps below to prepare the gMSA in the domain controller:
Create a domain user account. This user account/password will be saved as a Kubernetes secret and used to retrieve the gMSA password.
To create a GMSA account and grant permission to read the password for the gMSA account created in Step 2, run the following New-ADServiceAccount PowerShell command:
New-ADServiceAccount -Name "<gmsa account name>" -DnsHostName "<gmsa account name>.<domain name>.com" -ServicePrincipalNames "host/<gmsa account name>", "host/<gmsa account name>.<domain name>.com" -PrincipalsAllowedToRetrieveManagedPassword <username you created earlier>
-PrincipalsAllowedToRetrieveManagedPassword, specify the domain username you created earlier as shown in the example below:
New-ADServiceAccount -Name "WebApp01" -DnsHostName "WebApp01.akshcitest.com" -ServicePrincipalNames "host/WebApp01", "host/WebApp01.akshcitest.com" -PrincipalsAllowedToRetrieveManagedPassword "testgmsa"
Prepare the gMSA credential spec JSON file
To prepare the gMSA credential spec JSON file, follow the steps for creating a credential spec.
Configure gMSA for Windows pods and containers in the cluster
The Kubernetes community already supports domain joined Windows worker nodes for gMSA. Although you don't need to domain join a Windows worker node in AKS on Azure Stack HCI and Windows Server, there are other required configuration steps. These steps include installing the webhook, the custom resource definition (CRD), and the credential spec, as well as enabling role-based access control (RBAC role). The following steps use PowerShell commands that are built to help simplify the configuration process.
Before completing the steps below, make sure the AksHci PowerShell module is installed and
kubectl can connect to your cluster.
To install the webhook, run the following Install-AksHciGmsaWebhook PowerShell command:
Install-AksHciGMSAWebhook -Name <cluster name>
To validate that the webhook pod is successfully running, run the following command:
kubectl get pods -n kube-system | findstr gmsa
You should see one pod with the gmsa-webhook prefix that is running.
Create the secret object that stores the Active Directory user credential. Complete the configuration data below and save the settings into a file named secret.yaml.
apiVersion: v1 kind: Secret metadata: name: <secret-name> namespace: <secret under namespace other than the default> type: Opaque stringData: domain: <FQDN of the domain, for example: akshcitest.com> username: <domain user who can retrieve the gMSA password> password: <password>
Then, run the following command to create the secret object:
kubectl apply -f secret.yaml
If you create a secret under a namespace other than the default, remember to set the namespace of the deployment to the same namespace.
Use the Add-AksHciGMSACredentialSpec PowerShell cmdlet below to create the gMSA CRD, enable role-based access control (RBAC), and then assign the role to the service accounts to use a specific gMSA credential spec file. These steps are described in more detail in this Kubernetes article on Configure gMSA for Windows pods and containers.
Use the JSON credential spec as input for the following PowerShell command (parameters with asterisks * are mandatory):
Add-AksHciGMSACredentialSpec -Name <cluster name>* -credSpecFilePath <path to JSON credspec>* -credSpecName <name for credspec as the k8s GMSACredentialSpec object>* -secretName <name of secret>* -secretNamespace <namespace of secret> -serviceAccount <name of service account to bind to clusterrole> -clusterRoleName <name of clusterrole to use the credspec>* -overwrite
To view an example, see the following:
Add-AksHciGMSACredentialSpec -Name mynewcluster -credSpecFilePath .\credspectest.json -credSpecName credspec-mynewcluster -secretName mysecret -clusterRoleName clusterrole-mynewcluster
Deploy the application
Create the deployment YAML file using the following example settings. The required entries include
Add the service account:
serviceAccountName: default securityContext: windowsOptions: gmsaCredentialSpecName:
Add the credential spec object:
securityContext: windowsOptions: gmsaCredentialSpecName: <cred spec name>
Mount the secret for the deployment:
volumeMounts: - name: <volume name> mountPath: <vmount path> readOnly: true volumes: - name: <volume name> secret: secretName: <secret name> items: - key: username path: my-group/my-username
Add the IP address of the domain controller and domain name under dnsConfig:
dnsConfig: nameservers: - <IP address for domain controller> searches: - <domain>
Verify the container is working with gMSA
After you deploy the container, use the following steps to verify that it's working:
Get the container ID for your deployment by running the following command:
kubectl get pods
Open PowerShell and run the following command:
kubectl exec -it <container id> powershell
Once you are in the container, run the following:
Nltest /parentdomain Nltest /sc_verify:<domain>
Connection Status = 0 0x0 NERR_Success The command completed successfully.
This output shows that the computer has been authenticated by a domain controller, and a secure channel exists between the client computer and the domain controller.
Check if the container can obtain a valid Kerberos Ticket Granting Ticket (TGT).
klist get krbtgt
A ticket to krbtgt has been retrieved successfully
Clean up gMSA settings in the cluster
If you need to clean up gMSA settings, use the following uninstall steps.
Uninstall the credential spec
To uninstall, run the following Remove-AksHcigmsaCredentialSpec PowerShell command:
Remove-AksHciGMSACredentialSpec -Name <cluster name> -credSpecName <cred spec object name> -clusterRoleName <clusterrole object name> -serviceAccount <serviceaccount object name> -secretNamespace <namespace of the secret object>
To uninstall the webhook, run the following Uninstall-AksHciGMSAWebhook PowerShell command:
Uninstall-AksHciGMSAWebhook -Name <cluster name>