Use Azure RBAC for AKS hybrid clusters (preview)

Applies to: AKS on Azure Stack HCI, AKS on Windows Server

This article describes how to set up Azure RBAC on an AKS hybrid cluster to use Azure Active Directory (Azure AD) and Azure role assignments for authorization. Steps for creating the cluster are covered in Prerequisites.

For a conceptual overview of using Azure RBAC with Azure Arc-enabled Kubernetes clusters, see Azure RBAC on Azure Arc-enabled Kubernetes.

Important

These preview features are available on a self-service, opt-in basis. Previews are provided "as is" and "as available," and they're excluded from the service-level agreements and limited warranty. Azure Arc-enabled Kubernetes previews are partially covered by customer support on a best-effort basis.

Prerequisites

Before you deploy an AKS hybrid cluster with Azure Arc enabled, you must complete the following prerequisites.

Prepare your network

Configure the following network, proxy, and/or firewall settings:

• Configure the endpoints that must be accessible to connect a cluster to Azure Arc. For a list, see Meet network requirements.

• Allow the Graph endpoint in your proxy or firewall.

  For information about configuring a proxy server, see Proxy server settings.

Create the server app and client app

Register your server app and secret and your client app and secret by performing the following steps:

Note

These steps direct you to key tasks in Use Azure RBAC for Azure Arc-enabled Kubernetes clusters that are required to prepare for the Azure RBAC setup in AKS hybrid.

To do these steps, you must have the built-in Application Administrator role in Azure AD. For instructions, see Assign Azure AD roles to users.

1. Create a server application and shared secret.

2. Create a role assignment for the server application.

3. Create a client application. You refer to the client application when you use kubectl to connect within your network.

Grant permissions for users on the cluster

Assign roles to grant permissions to users of service principal names (SPNs) on the cluster. Use the az role assignment command.

To assign roles on an AKS hybrid cluster, you must have Owner permission on the subscription, resource group, or cluster.

The following example uses az role assignment to assign the Azure Arc Kubernetes Cluster Admin role to the resource group that contains the cluster. You can set the scope of the resource group before you create the cluster.

az role assignment create --role "Azure Arc Kubernetes Cluster Admin" --assignee xyz@contoso.com --scope /subscriptions/<subscription id>/resourceGroups/<resource group name>/providers/Microsoft.Kubernetes/connectedClusters/<resource name, aka name of AKS cluster>

In order to access the cluster via the connectedk8s proxy method (one of the options to communicate with the api-server), you must have the "Azure Arc Enabled Kubernetes Cluster User Role" scoped to the subscription, resource group, or cluster.

The following command assigns a role to a group instead of a specific user (see the previous example):

az role assignment create --assignee 00000000-0000-0000-0000-000000000000 --role "Azure Arc Kubernetes Cluster Admin" --scope $id

The assignee is the object ID of the Azure AD group.

For more information about the role, see this section.

To get the scope ID for the cluster or resource group, run the following commands, and use the "id":property:

az connectedk8s show -g <name of resource group>  
az connectedk8s show -n <name of cluster> -g <name of resource group>

For other examples, see az role assignment.

For information about pre-built Azure RBAC roles for Arc-enabled Kubernetes clusters, see Create role assignments for users to access a cluster. For a list of all available built-in roles, see Azure built-in roles.

Step 1: Create an SPN and assign permissions

Use an Azure service principal to configure an automation account with the permissions needed to create a target cluster with Azure RBAC enabled.

Creating a target cluster only requires limited privileges on the subscription. We recommend using the Kubernetes Cluster - Azure Arc Onboarding role. You can also use the Owner or Contributor role. For more information, see Azure built-in roles.

Use the az ad sp create-for-rbac command in Azure CLI to create the SPN and configure it with the needed permissions.

The following example assigns the Kubernetes Cluster - Azure Arc Onboarding role to the subscription. For more information, see the az ad sp command reference.

az ad sp create-for-rbac --role "Kubernetes Cluster - Azure Arc Onboarding" --scopes /subscriptions/<OID of the subscription ID> 

Important

The command output for az ad sp includes credentials that you must protect. Do not include these credentials in your code or check the credentials into your source control. For more information, see Create an Azure service principal.

For more information about creating an SPN and assigning it a role, see Create an Azure service principal.

Step 2: Create the credential object

To create a credential object to use with the SPN, open a PowerShell window, and run the following command:

$Credential = Get-Credential

This command prompts for a password.

To automate creation of the credential object, without requiring manual password entry, see Get-Credential, Example 4. The script includes a plaintext credential, which might violate security standards in some enterprises.

Step 3: Create an Azure RBAC-enabled AKS hybrid cluster

You can create an Azure RBAC-enabled cluster using an SPN (Option A) or create the cluster interactively (Option B).

Option A: Create Azure RBAC-enabled AKS hybrid target cluster using an SPN

To create an AKS hybrid target cluster with Azure RBAC enabled using an SPN:

1. If you haven't already created an SPN to use with the target cluster, create the SPN now.

2. Note the SPN created is for one time use when creating the cluster and doesn't require managing passwords

3. Open a PowerShell window on the Azure HCI node or Windows server where you deploy the cluster, and run the following command:

   New-AksHciCluster -name "<cluster name>"  -enableAzureRBAC -resourceGroup "<resource group name>" -subscriptionID "<subscription ID>" -tenantId "<tenant ID>" -credential $Credential -location "eastus" -appId $SERVER_APP_ID -appSecret $SERVER_APP_SECRET -aadClientId $CLIENT_APP_ID -nodePoolName <name of node pool> 
   

Option B: Create Azure RBAC-enabled AKS hybrid target cluster interactively

If you prefer to create your Azure RBAC-enabled target cluster interactively, follow these steps:

1. Open a PowerShell window on the Azure HCI node or Windows server where you deploy the cluster.

2. Sign in to Azure by running the following command. using the connect-azaccount -deviceauth command.

   connect-azaccount -deviceauth
   

   The command prompts for authentication via the device code flow.

3. Set the subscription context to the subscription where the target cluster is to be created:

   az account set --subscription "subscriptionName"
   

4. Create the AKS hybrid target cluster, with Azure RBAC enabled:

   New-AksHciCluster -name "<cluster name>"  -enableAzureRBAC -resourceGroup "<name of resource group>"  -location "eastus" -appId $SERVER_APP_ID -appSecret $SERVER_APP_SECRET -aadClientId $CLIENT_APP_ID -nodePoolName <name of node pool> 
   

Step 4: Connect to AKS hybrid cluster via Azure RBAC

The Azure RBAC setup on the AKS cluster is now complete. To test your Azure RBAC setup, connect to the AKS cluster. Azure RBAC authenticates the connections.

The procedures in this section use the connectedk8s proxy method to connect to an AKS cluster and connect to an AKS cluster over a private network.

Connect to AKS hybrid cluster over the internet using the connectedk8s proxy method

Use the connectedk8s proxy method to send an authentication/authorization request from anywhere on the internet. When you use this method, you're limited to 200 groups.

To connect to an AKS hybrid cluster using the connectedk8s proxy method, do the following steps:

1. Open an Azure CLI window, and use az login to connect to Azure. For more information, see Sign in with Azure CLI.

2. Set the subscription for your Azure account to the subscription you used to create the AKS hybrid cluster if needed:

   az account set -subscription "<mySubscription>" 
   

3. Confirm that you're connected to Azure:

   az account show
   

4. Start the proxy process:

   az connectedk8s proxy -n <cluster name> -g <resource group name>
   

5. Make sure authentication is working correctly by sending requests to the cluster. Leaving open the terminal that you connected from, open another tab, and send the requests to the cluster. You should get responses based on your Azure RBAC configuration.

6. Press Ctrl+C to close the connectedk8s proxy connection.

Connect to AKS hybrid cluster over a private network

When you connect to an AKS hybrid cluster over a private network, there's no limit the on number of groups you can use.

To retrieve the Azure AD kubeconfig log into and on-premises machine (for example, an HCI cluster), generate the Azure AD kubeconfig using the following command. You can distribute the Azure AD kubeconfig to users that connect from their client machine. The Azure AD kubeconfig doesn't contain any secrets.

To connect to an AKS hybrid cluster over a private network, perform the following steps:

1. Download the kubeconfig file:

   Get-AksHciCredential -Name <cluster name> -aadauth
   

2. Start sending requests to AKS API server by running the kubectl command api-server. You are prompted for your Azure AD credentials.

   You might get a warning message, but you can ignore it.

Update to the kubelogin authentication plugin

Note

The information in this section applies to AKS hybrid version 1.0.17.10310 and later. See the release notes for version information.

To provide authentication tokens for communicating with AKS hybrid clusters, Kubectl clients require an authentication plugin.

To generate a kubeconfig file that requires the Azure kubelogin.exe binary authentication plugin, run the following PowerShell command:

Get-AksHciCredential -Name <cluster name> -aadauth

This command also downloads the kubelogin.exe binary. To find the location of the kubelogin.exe file, run the following command:

$workingdir = (Get-AksHciConfig).Akshci.installationPackageDir

This command returns the path to which kubelogin.exe is downloaded. Copy the kubelogin.exe file to your HCI node or client machine. For HCI, copy the file to the path as described in the following example. For a client machine, copy the executable to your client machine and add it to your path. For example:

cp $workingdir\kubelogin.exe "c:\program files\akshci"

Alternatively, to download kubelogin.exe to your client machine, you can run the following command:

wget https://github.com/Azure/kubelogin/releases/download/v0.0.26/kubelogin-win-amd64.zip -OutFile kubelogin-win-amd64.zip

For more information about how to convert to the kubelogin authentication plugin, see the Azure kubelogin page on GitHub.

Next steps

• Learn more about SPNs