Events
Mar 17, 9 PM - Mar 21, 10 AM
Join the meetup series to build scalable AI solutions based on real-world use cases with fellow developers and experts.
Register nowThis browser is no longer supported.
Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.
This article shows you how to securely scale your applications with the Kubernetes Event-driven Autoscaling (KEDA) add-on and workload identity on Azure Kubernetes Service (AKS).
Important
Your cluster Kubernetes version determines what KEDA version will be installed on your AKS cluster. To see which KEDA version maps to each AKS version, see the AKS managed add-ons column of the Kubernetes component version table.
For GA Kubernetes versions, AKS offers full support of the corresponding KEDA minor version in the table. Kubernetes preview versions and the latest KEDA patch are partially covered by customer support on a best-effort basis. As such, these features aren't meant for production use. For more information, see the following support articles:
Create a resource group using the az group create
command. Make sure you replace the placeholder values with your own values.
LOCATION=<azure-region>
RG_NAME=<resource-group-name>
az group create --name $RG_NAME --location $LOCATION
Create an AKS cluster with the KEDA add-on, workload identity, and OIDC issuer enabled using the az aks create
command with the --enable-workload-identity
, --enable-keda
, and --enable-oidc-issuer
flags. Make sure you replace the placeholder value with your own value.
AKS_NAME=<cluster-name>
az aks create \
--name $AKS_NAME \
--resource-group $RG_NAME \
--enable-workload-identity \
--enable-oidc-issuer \
--enable-keda \
--generate-ssh-keys
Validate the deployment was successful and make sure the cluster has KEDA, workload identity, and OIDC issuer enabled using the az aks show
command with the --query
flag set to "[workloadAutoScalerProfile, securityProfile, oidcIssuerProfile]"
.
az aks show \
--name $AKS_NAME \
--resource-group $RG_NAME \
--query "[workloadAutoScalerProfile, securityProfile, oidcIssuerProfile]"
Connect to the cluster using the az aks get-credentials
command.
az aks get-credentials \
--name $AKS_NAME \
--resource-group $RG_NAME \
--overwrite-existing
Create an Azure Service Bus namespace using the az servicebus namespace create
command. Make sure to replace the placeholder value with your own value.
SB_NAME=<service-bus-name>
SB_HOSTNAME="${SB_NAME}.servicebus.windows.net"
az servicebus namespace create \
--name $SB_NAME \
--resource-group $RG_NAME \
--disable-local-auth
Create an Azure Service Bus queue using the az servicebus queue create
command. Make sure to replace the placeholder value with your own value.
SB_QUEUE_NAME=<service-bus-queue-name>
az servicebus queue create \
--name $SB_QUEUE_NAME \
--namespace $SB_NAME \
--resource-group $RG_NAME
Create a managed identity using the az identity create
command. Make sure to replace the placeholder value with your own value.
MI_NAME=<managed-identity-name>
MI_CLIENT_ID=$(az identity create \
--name $MI_NAME \
--resource-group $RG_NAME \
--query "clientId" \
--output tsv)
Get the OIDC issuer URL using the az aks show
command with the --query
flag set to oidcIssuerProfile.issuerUrl
.
AKS_OIDC_ISSUER=$(az aks show \
--name $AKS_NAME \
--resource-group $RG_NAME \
--query oidcIssuerProfile.issuerUrl \
--output tsv)
Create a federated credential between the managed identity and the namespace and service account used by the workload using the az identity federated-credential create
command. Make sure to replace the placeholder value with your own value.
FED_WORKLOAD=<federated-credential-workload-name>
az identity federated-credential create \
--name $FED_WORKLOAD \
--identity-name $MI_NAME \
--resource-group $RG_NAME \
--issuer $AKS_OIDC_ISSUER \
--subject system:serviceaccount:default:$MI_NAME \
--audience api://AzureADTokenExchange
Create a second federated credential between the managed identity and the namespace and service account used by the keda-operator using the az identity federated-credential create
command. Make sure to replace the placeholder value with your own value.
FED_KEDA=<federated-credential-keda-name>
az identity federated-credential create \
--name $FED_KEDA \
--identity-name $MI_NAME \
--resource-group $RG_NAME \
--issuer $AKS_OIDC_ISSUER \
--subject system:serviceaccount:kube-system:keda-operator \
--audience api://AzureADTokenExchange
Get the object ID for the managed identity using the az identity show
command with the --query
flag set to "principalId"
.
MI_OBJECT_ID=$(az identity show \
--name $MI_NAME \
--resource-group $RG_NAME \
--query "principalId" \
--output tsv)
Get the Service Bus namespace resource ID using the az servicebus namespace show
command with the --query
flag set to "id"
.
SB_ID=$(az servicebus namespace show \
--name $SB_NAME \
--resource-group $RG_NAME \
--query "id" \
--output tsv)
Assign the Azure Service Bus Data Owner role to the managed identity using the az role assignment create
command.
az role assignment create \
--role "Azure Service Bus Data Owner" \
--assignee-object-id $MI_OBJECT_ID \
--assignee-principal-type ServicePrincipal \
--scope $SB_ID
After creating the federated credential for the keda-operator
ServiceAccount, you will need to manually restart the keda-operator
pods to ensure Workload Identity environment variables are injected into the pod.
kubectl rollout restart deploy keda-operator -n kube-system
Confirm the keda-operator pods restart
kubectl get pod -n kube-system -lapp=keda-operator -w
Once you've confirmed the keda-operator pods have finished rolling hit Ctrl+c
to break the previous watch command then confirm the Workload Identity environment variables have been injected.
KEDA_POD_ID=$(kubectl get po -n kube-system -l app.kubernetes.io/name=keda-operator -ojsonpath='{.items[0].metadata.name}')
kubectl describe po $KEDA_POD_ID -n kube-system
You should see output similar to the following under Environment.
---
AZURE_CLIENT_ID:
AZURE_TENANT_ID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx
AZURE_FEDERATED_TOKEN_FILE: /var/run/secrets/azure/tokens/azure-identity-token
AZURE_AUTHORITY_HOST: https://login.microsoftonline.com/
---
Deploy a KEDA TriggerAuthentication resource that includes the User-Assigned Managed Identity's Client ID.
kubectl apply -f - <<EOF
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
name: azure-servicebus-auth
namespace: default # this must be same namespace as the ScaledObject/ScaledJob that will use it
spec:
podIdentity:
provider: azure-workload
identityId: $MI_CLIENT_ID
EOF
Note
With the TriggerAuthentication in place, KEDA will be able to authenticate via workload identity. The keda-operator
Pods use the identityId
to authenticate against Azure resources when evaluating scaling triggers.
At this point everything is configured for scaling with KEDA and Microsoft Entra Workload Identity. We will test this by deploying producer and consumer workloads.
Create a new ServiceAccount for the workloads.
kubectl apply -f - <<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
annotations:
azure.workload.identity/client-id: $MI_CLIENT_ID
name: $MI_NAME
EOF
Deploy a Job to publish 100 messages.
kubectl apply -f - <<EOF
apiVersion: batch/v1
kind: Job
metadata:
name: myproducer
spec:
template:
metadata:
labels:
azure.workload.identity/use: "true"
spec:
serviceAccountName: $MI_NAME
containers:
- image: ghcr.io/azure-samples/aks-app-samples/servicebusdemo:latest
name: myproducer
resources: {}
env:
- name: OPERATION_MODE
value: "producer"
- name: MESSAGE_COUNT
value: "100"
- name: AZURE_SERVICEBUS_QUEUE_NAME
value: $SB_QUEUE_NAME
- name: AZURE_SERVICEBUS_HOSTNAME
value: $SB_HOSTNAME
restartPolicy: Never
EOF
Now that we have published messages to the Azure Service Bus queue, we will deploy a ScaledJob to consume the messages. This ScaledJob will use the KEDA TriggerAuthentication resource to authenticate against the Azure Service Bus queue using the workload identity and scale out every 10 messages.
Deploy a ScaledJob resource to consume the messages. The scale trigger will be configured to scale out every 10 messages. The KEDA scaler will create 10 jobs to consume the 100 messages.
kubectl apply -f - <<EOF
apiVersion: keda.sh/v1alpha1
kind: ScaledJob
metadata:
name: myconsumer-scaledjob
spec:
jobTargetRef:
template:
metadata:
labels:
azure.workload.identity/use: "true"
spec:
serviceAccountName: $MI_NAME
containers:
- image: ghcr.io/azure-samples/aks-app-samples/servicebusdemo:latest
name: myconsumer
env:
- name: OPERATION_MODE
value: "consumer"
- name: MESSAGE_COUNT
value: "10"
- name: AZURE_SERVICEBUS_QUEUE_NAME
value: $SB_QUEUE_NAME
- name: AZURE_SERVICEBUS_HOSTNAME
value: $SB_HOSTNAME
restartPolicy: Never
triggers:
- type: azure-servicebus
metadata:
queueName: $SB_QUEUE_NAME
namespace: $SB_NAME
messageCount: "10"
authenticationRef:
name: azure-servicebus-auth
EOF
Note
ScaledJob creates a Kubernetes Job resource whenever a scaling event occurs and thus a Job template needs to be passed in when creating the resource. As new Jobs are created, Pods will be deployed with workload identity bits to consume messages.
Verify the KEDA scaler worked as intended.
kubectl describe scaledjob myconsumer-scaledjob
You should see events similar to the following.
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal KEDAScalersStarted 10m scale-handler Started scalers watch
Normal ScaledJobReady 10m keda-operator ScaledJob is ready for scaling
Warning KEDAScalerFailed 10m scale-handler context canceled
Normal KEDAJobsCreated 10m scale-handler Created 10 jobs
After you verify that the deployment is successful, you can clean up the resources to avoid incurring Azure costs.
Delete the Azure resource group and all resources in it using the [az group delete
][az-group-delete] command.
az group delete --name $RG_NAME --yes --no-wait
This article showed you how to securely scale your applications using the KEDA add-on and workload identity in AKS.
For information on KEDA troubleshooting, see Troubleshoot the Kubernetes Event-driven Autoscaling (KEDA) add-on.
To learn more about KEDA, see the upstream KEDA docs.
Azure Kubernetes Service feedback
Azure Kubernetes Service is an open source project. Select a link to provide feedback:
Events
Mar 17, 9 PM - Mar 21, 10 AM
Join the meetup series to build scalable AI solutions based on real-world use cases with fellow developers and experts.
Register now