Kubernetes / AKS cluster requirments for Azure ML Endpoints

Attila 6

Hello,

I'm trying to experiment with Azure ML Endpoints deployed to a Azure Kubernetes Service (AKS) cluster.

Right now I'm able to attach AKS cluster(s) to Azure ML, from Compute / Inference Clusters.

However, the no Kubernetes clusters are showing up nor in Endpoint Deployments, neither in the Attached Computes (see the screenshots bellow).

I would like to now if there are any special requirements / additional steps needed for AKS cluster used with Azure ML endpoints. The AKS cluster I'm using is 3 nodes, each with 4 vCPU and 16 GB RAM.

Endpoint(s):

Attached Compute(s):

Thanks!

romungi-MSFT 45,721 Reputation points Microsoft Employee

2022-08-17T10:41:36.627+00:00

@Attila I think the reason that the cluster is not showing up because of the limitations or pre-requisites that are required to use the Kubernetes or Arc Kubernetes clusters for endpoint deployments.
Could you please try to check if you could add the AzureML extension to the cluster by following the steps in this document?
Attila 6 Reputation points

2022-08-18T10:02:45.453+00:00

Hmm, I just assumed the AzureML extension would be automatically installed for AKS clusters. There seems to be no Settings / Extensions option for AKS in Azure Portal (like for Arc).

I tried to install the AzureML K8S extension from the CLI:

az k8s-extension create --name aks-azure-ml --extension-type Microsoft.AzureML.Kubernetes --config enableTraining=True enableInference=True inferenceRouterServiceType=LoadBalancer allowInsecureConnections=True inferenceLoadBalancerHA=False --cluster-type managedClusters --cluster-name aks-azure-ml --resource-group azure-ml --scope cluster

but I get authorization error:

(ExtensionOperationFailed) The extension operation failed with the following error: Request failed to https://management.azure.com/subscriptions/ec0bd9e0-..../resourceGroups/azure-ml/providers/Microsoft.ContainerService/managedclusters/aks-azure-ml/extensionaddons/aks-azure-ml?api-version=2021-03-01. Error code: Unauthorized. Reason: Unauthorized.{"error":{"code":"InvalidAuthenticationToken","message":"The received access token is not valid: at least one of the claims 'puid' or 'altsecid' or 'oid' should be present. If you are accessing as application please make sure service principal is properly created in the tenant."}}.

Is there is start to end tutorial on how to this? I guess, some additional steps are needed...

Thanks!
romungi-MSFT 45,721 Reputation points Microsoft Employee

2022-08-18T14:33:03.577+00:00
I tried the steps in the document and the above command is what I had to use to enable the cluster for training and inference. I think in your case the AKS is running with a service principal that is causing it to fail. You could try to use system assigned managed identity with the cluster or create a new AKS cluster with the minimum node requirement and install the extension with this command using azure cli.

az k8s-extension create --name <new_name> --extension-type Microsoft.AzureML.Kubernetes --config enableTraining=True enableInference=True inferenceRouterServiceType=LoadBalancer allowInsecureConnections=True inferenceLoadBalancerHA=False --cluster-type managedClusters --cluster-name <your_aks_cluster> --resource-group <your_resource_group> --scope cluster

Once this is setup, add the attached compute from the attached compute tab on ml.azure.com by selecting the AKS cluster and there is an option to assign the MI from this screen too. After creating this successfully, the AKS cluster or this attached compute is available for deploying a real time endpoint.
Attila 6 Reputation points

2022-08-23T16:33:35.18+00:00
Hi @romungi-MSFT ,

The AKS Clusters I'm using are all were created with "Assigned Managed Identity".

I tried to run the az k8s-extension create ... both from Cloud Shell and locally. But, I think something goes wrong with the authentication / authorization part, as the command always fails with the same error (see above).

I even tried to using a newly created managed identity power-user, with full "Owner" role on the subscription:

$ az login --identity -u /subscriptions/<ec0bd9e0-subscription-id>/resourcegroups/azure-ml/providers/Microsoft.ManagedIdentity/userAssignedIdentities/power-user $ az account set --subscription ec0bd9e0-... $ az aks get-credentials --resource-group azure-ml --name aks-test-ml-compute-3 $ az k8s-extension create --name aks-test-ml-compute-3-ext --extension-type Microsoft.AzureML.Kubernetes --config enableTraining=True enableInference=True inferenceRouterServiceType=LoadBalancer allowInsecureConnections=True inferenceLoadBalancerHA=False --cluster-type managedClusters --cluster-name aks-test-ml-compute-3 --resource-group azure-ml --scope cluster --verbose --debug

Here are the debug logs from the az k8s-extension create command ran with the --debug flag:
https://www.dropbox.com/scl/fi/l6g60ssqo96q19xuflgw9/az-k8s-extension-error-log.paper?dl=0&rlkey=gl3p478r5rsnu6rl5bpzlpfkc

Could you please take a look?

Thanks!

1 answer

romungi-MSFT 45,721 Reputation points Microsoft Employee

2022-08-24T05:29:49.9+00:00
@Attila I feel the error is most likely an issue with the AKS cluster setup. I see the following in the log:

"aksAssignedIdentity":null

I don't have a lot of experience with AKS setup issues, but I feel this could be the issue here. As per AKS documentation there is an option to update to system assigned managed identity.

az aks update -g <RGName> -n <AKSName> --enable-managed-identity

If this still does not work I think it would be easier to review the setup through a support case so the service team could advise what could be wrong with your subscription.

If an answer is helpful, please click on or upvote which might help other community members reading this thread.
Please sign in to rate this answer.
Attila 6 Reputation points

2022-08-24T20:12:17.14+00:00

@romungi-MSFT , running the update command:

$ az aks update -g <RGName> -n <AKSName> --enable-managed-identity

did the trick. Thank! :)

romungi-MSFT 45,721 Reputation points Microsoft Employee

2022-09-07T10:22:13.94+00:00

@Attila That's great!! It would be helpful if you could accept my response as an answer, this would help other users with similar scenario to find the answer sooner. Thanks!!
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Kubernetes / AKS cluster requirments for Azure ML Endpoints

1 answer

Your answer