Outbound connection failed when connecting k8s cluster

Question

I'm trying to register an existing k8s cluster with azure arc. I've run the following command on one of the k8s nodes:

az connectedk8s connect -g $ARC_RG_NAME -n $ARC_CLUSTER_NAME -l "West Europe"

And get the following output:

~$ az connectedk8s connect -g $ARC_RG_NAME -n $ARC_CLUSTER_NAME -l "West Europe"
This operation might take a while...
The outbound network connectivity check has failed for the endpoint - https://westeurope.obo.arc.azure.com:8084/
This will affect the "cluster-connect" feature. If you are planning to use "cluster-connect" functionality , please ensure outbound connectivity to the above endpoint.
Error: We found an issue with outbound network connectivity from the cluster to the endpoints required for onboarding.
Please ensure to meet the following network requirements 'https://learn.microsoft.com/en-us/azure/azure-arc/kubernetes/network-requirements?tabs=azure-cloud'
If your cluster is behind an outbound proxy server, please ensure that you have passed proxy parameters during the onboarding of your cluster.
For more details visit 'https://docs.microsoft.com/en-us/azure/azure-arc/kubernetes/quickstart-connect-cluster?tabs=azure-cli#connect-using-an-outbound-proxy-server'
The pre-check result logs logs have been saved at this path: /home/user/.azure/pre_onboarding_check_logs/k3sArc-cluster-Fri-Jun-21-14.43.27-2024.
These logs can be attached while filing a support ticket for further assistance.
One or more pre-onboarding diagnostic checks failed and hence not proceeding with             cluster onboarding. Please resolve them and try onboarding again.

The content of outbound_network_connectivity_check_for_cluster_connect.txt has the following:

Response code 000
Outbound connectivity failed for the endpoint:https://westeurope.obo.arc.azure.com:8084/ ,this is an optional endpoint needed for cluster-connect feature.

And attempting to go to https://westeurope.obo.arc.azure.com:8084/ in a browser shows a 500 error.

I don't have an outbound proxy, and i have excluded the server from any and all firewall rules (there isn't many, but i turned it off just to check).

I ran a curl commend to check from the same server:

curl -Iv https://westeurope.obo.arc.azure.com:8084/
*   Trying 20.61.96.184:8084...
* Connected to westeurope.obo.arc.azure.com (20.61.96.184) port 8084 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=WA; L=Redmond; O=Microsoft Corporation; CN=westeurope.obo.arc.azure.com
*  start date: Apr  4 07:36:47 2024 GMT
*  expire date: Mar 30 07:36:47 2025 GMT
*  subjectAltName: host "westeurope.obo.arc.azure.com" matched cert's "westeurope.obo.arc.azure.com"
*  issuer: C=US; O=Microsoft Corporation; CN=Microsoft Azure RSA TLS Issuing CA 04
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* Using Stream ID: 1 (easy handle 0x5584959fc990)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> HEAD / HTTP/2
> Host: westeurope.obo.arc.azure.com:8084
> user-agent: curl/7.81.0
> accept: */*
>
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 500
HTTP/2 500
< date: Fri, 21 Jun 2024 13:25:08 GMT
date: Fri, 21 Jun 2024 13:25:08 GMT
< server: Kestrel
server: Kestrel
< content-length: 0
content-length: 0

<

* Connection #0 to host westeurope.obo.arc.azure.com left intact

Which seem to imply the ssl scert is find but the server is returning a 500 error.