Use cluster connect to securely connect to Azure Arc-enabled Kubernetes clusters
With cluster connect, you can securely connect to Azure Arc-enabled Kubernetes clusters from anywhere without requiring any inbound port to be enabled on the firewall.
Access to the apiserver
of the Azure Arc-enabled Kubernetes cluster enables the following scenarios:
- Interactive debugging and troubleshooting.
- Cluster access to Azure services for custom locations and other resources created on top of it.
Before you begin, review the conceptual overview of the cluster connect feature.
Prerequisites
An Azure account with an active subscription. Create an account for free.
An existing Azure Arc-enabled Kubernetes connected cluster.
- If you haven't connected a cluster yet, use our quickstart.
- Upgrade your agents to the latest version.
Enable these endpoints for outbound access:
Endpoint Port *.servicebus.windows.net
443 guestnotificationservice.azure.com
,*.guestnotificationservice.azure.com
443 Note
To translate the
*.servicebus.windows.net
wildcard into specific endpoints, use the command\GET https://guestnotificationservice.azure.com/urls/allowlist?api-version=2020-01-01&location=<location>
. Within this command, the region must be specified for the<location>
placeholder.
To get the region segment of a regional endpoint, remove all spaces from the Azure region name. For example, East US 2 region, the region name is eastus2
.
For example: *.<region>.arcdataservices.com
should be *.eastus2.arcdataservices.com
in the East US 2 region.
To see a list of all regions, run this command:
az account list-locations -o table
Get-AzLocation | Format-Table
Install the latest version of the
connectedk8s
Azure CLI extension:az extension add --name connectedk8s
If you've already installed the
connectedk8s
extension, update the extension to the latest version:az extension update --name connectedk8s
Replace the placeholders and run the below command to set the environment variables used in this document:
CLUSTER_NAME=<cluster-name> RESOURCE_GROUP=<resource-group-name> ARM_ID_CLUSTER=$(az connectedk8s show -n $CLUSTER_NAME -g $RESOURCE_GROUP --query id -o tsv)
Set up authentication
On the existing Arc-enabled cluster, create the ClusterRoleBinding with either Microsoft Entra authentication or service account token.
Microsoft Entra authentication option
Get the
objectId
associated with your Microsoft Entra entity. If you are using a single user account, get the user principal name (UPN) associated with your Microsoft Entra entity.- For a Microsoft Entra group account:
AAD_ENTITY_ID=$(az ad signed-in-user show --query id -o tsv)
For a Microsoft Entra single user account:
AAD_ENTITY_ID=$(az ad signed-in-user show --query userPrincipalName -o tsv)
For a Microsoft Entra application:
AAD_ENTITY_ID=$(az ad sp show --id <id> --query id -o tsv)
Authorize the entity with appropriate permissions.
If you're using Kubernetes native ClusterRoleBinding or RoleBinding for authorization checks on the cluster, with the
kubeconfig
file pointing to theapiserver
of your cluster for direct access, you can create one mapped to the Microsoft Entra entity (service principal or user) that needs to access this cluster. For example:kubectl create clusterrolebinding demo-user-binding --clusterrole cluster-admin --user=$AAD_ENTITY_ID
If you're using Azure RBAC for authorization checks on the cluster, you can create an applicable Azure role assignment mapped to the Microsoft Entra entity. For example:
az role assignment create --role "Azure Arc Kubernetes Viewer" --assignee $AAD_ENTITY_ID --scope $ARM_ID_CLUSTER az role assignment create --role "Azure Arc Enabled Kubernetes Cluster User Role" --assignee $AAD_ENTITY_ID --scope $ARM_ID_CLUSTER
Service account token authentication option
With the
kubeconfig
file pointing to theapiserver
of your Kubernetes cluster, run this command to create a service account. This example creates the service account in the default namespace, but you can substitute any other namespace fordefault
.kubectl create serviceaccount demo-user -n default
Create ClusterRoleBinding to grant this service account the appropriate permissions on the cluster. If you used a different namespace in the first command, substitute it here for
default
.kubectl create clusterrolebinding demo-user-binding --clusterrole cluster-admin --serviceaccount default:demo-user
Create a service account token:
kubectl apply -f - <<EOF apiVersion: v1 kind: Secret metadata: name: demo-user-secret annotations: kubernetes.io/service-account.name: demo-user type: kubernetes.io/service-account-token EOF
TOKEN=$(kubectl get secret demo-user-secret -o jsonpath='{$.data.token}' | base64 -d | sed 's/$/\n/g')
Get the token to output to console
echo $TOKEN
Access your cluster from a client device
Now you can access the cluster from a different client. Run the following steps on another client device.
Sign in using either Microsoft Entra authentication or service account token authentication.
Get the cluster connect
kubeconfig
needed to communicate with the cluster from anywhere (from even outside the firewall surrounding the cluster), based on the authentication option used:If using Microsoft Entra authentication:
az connectedk8s proxy -n $CLUSTER_NAME -g $RESOURCE_GROUP
If using service account token authentication:
az connectedk8s proxy -n $CLUSTER_NAME -g $RESOURCE_GROUP --token $TOKEN
Note
This command will open the proxy and block the current shell.
In a different shell session, use
kubectl
to send requests to the cluster:kubectl get pods -A
You should now see a response from the cluster containing the list of all pods under the default
namespace.
Known limitations
Use az connectedk8s show
to check your Arc-enabled Kubernetes agent version.
When making requests to the Kubernetes cluster, if the Microsoft Entra entity used is a part of more than 200 groups, you might see the following error:
You must be logged in to the server (Error:Error while retrieving group info. Error:Overage claim (users with more than 200 group membership) is currently not supported.
This is a known limitation. To get past this error:
- Create a service principal, which is less likely to be a member of more than 200 groups.
- Sign in to Azure CLI with the service principal before running the
az connectedk8s proxy
command.
Next steps
- Set up Microsoft Entra RBAC on your clusters.
- Deploy and manage cluster extensions.
Feedback
https://aka.ms/ContentUserFeedback.
Coming soon: Throughout 2024 we will be phasing out GitHub Issues as the feedback mechanism for content and replacing it with a new feedback system. For more information see:Submit and view feedback for