Azure Arc K8s cluster error: ClusterUnreachable: The underlying cluster is unreachable.

Question

Azure Arc K8s cluster error: ClusterUnreachable: The underlying cluster is unreachable.

Jake K 56

I have connected my on-premise K8s cluster via Azure Arc. The setup seems to work ok as I can paste in the bearer token and see all namespaces, pods, etc.

Now I want to use this arc-enabled K8s cluster as a target within my Azure Machine Learning Workspace.

I added it as attached compute in the workspace. Allocation Stats says "Succeeded".

But at the top, in a yellow information bar it says: "ClusterUnreachable: The underlying cluster is unreachable."

As a result I am unable to deploy any endpoints to my on-premise K8s cluster.

What might the issue be?

Capture

Rohit Mungi 49,126 Reputation points Microsoft Employee Moderator

2023-05-15T06:48:08.64+00:00

@Jake K Did you deploy the Azure Machine Learning extension on an AKS or Arc Kubernetes cluster as mentioned here? Also, some of the limitations of this deployment might cause the cluster to become unreachable. Could you also review the same?
Rohit Mungi 49,126 Reputation points Microsoft Employee Moderator

2023-05-18T17:05:56.1133333+00:00

@Jake K Did you get a chance to check if the missing extension and limitations caused the cluster to become unreachable?
Jake K 56 Reputation points

2023-05-18T17:15:07.5566667+00:00

@Rohit Mungi - sorry for the late response! Thanks for the recommendation to check! The k8s extension is installed and I dont believe the limitations apply in my environment. I am going to try this on another cluster that is networked differently and see if the behavior there is different. Will let you know.
Jake K 56 Reputation points

2023-05-26T02:18:52.85+00:00

I did managed to get this error to go away. Our firewall was blocking some traffic and this resulted in the failure.

I do have a follow-up question: How does the Arc object in Azure communicate with the on-premise K8s cluster? Also what credentials is it using for this? For example, lets say I need to deploy a workload to the on-premise cluster from Azure, how is it targeting the on-premise cluster behind the firewall? And what account is it doing it with?
Jake K 56 Reputation points

2023-05-26T02:22:28.71+00:00

It turned out this was being caused by firewall traffic that was blocked.
A follow up question - how does the Azure Arc resource communicate with the on-premise cluster? Also what account does it use for this?
I am trying to deploy a ML scoring script to the on-premise cluster using the Attached K8s option. But I am still unsure on how the deployment would work considering that there are no credentials that I provided when connecting the on-premise cluster to Azure Arc