Azure Arc
A Microsoft cloud service that enables deployment of Azure services across hybrid and multicloud environments.
322 questions
Azure Arc-Enabled Kubernetes kube-aad-proxy pod fails with mount error
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(128, 32%, 22%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3ERT%3C/text%3E%3C/svg%3E)
Ryan Taylor
5
Reputation points
I am experiencing an issue where the kube-aad-proxy pod moves to CrashLoopBackOff state after repeated Failed starts. The problem is caused by the pod's inability to find the secret to mount the volume. I have read every article and tried to delete and re-create the pod multiple times, but the issue persists. I have even tried different versions of helm without success. The only warning in 'kubectl describe pod kube-aad-proxy-6f9ccb597d-2nrn6 -n azure-arc' is related to a mount, which requires a tls key and crt. I verified that the secret exists. The warning message is as follows:
MountVolume.SetUp failed for volume "kube-aad-proxy-tls" : secret "kube-aad-proxy-certificate" not found
{count} votes
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(60.8, 4%, 15%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EKM%3C/text%3E%3C/svg%3E)
kobulloc-MSFT 23,491 Reputation points Microsoft Employee2024-02-02T07:12:38.1566667+00:00 Hello, @Ryan Taylor !
Can you provide more information about the troubleshooting steps that you've already taken? A lot of the initial suggestions are confirming that everything is configured correctly:
- What namespace is the secret in?
- What permissions does the secret have?
- What is the volume mount path?
- Is there anything in the pod's logs?
From the official documentation:
- Troubleshoot platform issues for Azure Arc-enabled Kubernetes clusters
- Diagnose connection issues for Azure Arc-enabled Kubernetes clusters
One last possibility is a known issue with kube-proxy CrashLoopBackOff that may apply.
-
Ryan Taylor 5 Reputation points
2024-02-02T20:18:10.3166667+00:00 deleted
-
Ryan Taylor 5 Reputation points
2024-02-02T20:28:41.91+00:00 deleted
-
Ryan Taylor 5 Reputation points
2024-02-02T20:51:47.76+00:00 - What namespace is the secret in?
metadata: creationTimestamp: "2024-02-02T20:03:51Z" name: kube-aad-proxy-certificate namespace: azure-arc ownerReferences: - apiVersion: clusterconfig.azure.com/v1beta1 kind: ArcCertificate name: kube-aad-proxy- What permissions does the secret have?
- Where do I locate the permissions a secret has?
- What is the volume mount path?
Volumes: kube-aad-proxy-tls: Type: Secret (a volume populated by a Secret) SecretName: kube-aad-proxy-certificate Optional: false- Is there anything in the pod's logs?
- Failed to create authenticator. Error:Get "https://sts.windows.net/REMOVED/.well-known/openid-configuration": tls: failed to verify certificate: x509: certificate is valid for *.REMOVED.com, REMOVED.com, not sts.windows.net\nfailed to create provider for azure\ngithub.com/azure-core/ClusterConfigurationAgent/kube-aad-proxy/pkg/auth.NewAADAuthenticator\n\t/mnt/vss/_work/1/s/gopath/src/github.com/azure-core/ClusterConfigurationAgent/kube-aad-proxy/pkg/auth/aadAuthenticator.go:89\ngithub.com/azure-core/ClusterConfigurationAgent/kube-aad-proxy/pkg/server.(*Server).ListenAndServe\n\t/mnt/vss/_work/1/s/gopath/src/github.com/azure-core/ClusterConfigurationAgent/kube-aad-proxy/pkg/server/server.go:137\ngithub.com/azure-core/ClusterConfigurationAgent/kube-aad-proxy/cmd.NewRunCmd.func1\n\t/mnt/vss/_work/1/s/gopath/src/github.com/azure-core/ClusterConfigurationAgent/kube-aad-proxy/cmd/run.go:33\ngithub.com/spf13/cobra.(*Command).execute\n\t/mnt/vss/_work/1/s/gopath/pkg/mod/github.com/spf13/cobra@v1.6.0/command.go:920\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/mnt/vss/_work/1/s/gopath/pkg/mod/github.com/spf13/cobra@v1.6.0/command.go:1040\ngithub.com/spf13/cobra.(*Command).Execute\n\t/mnt/vss/_work/1/s/gopath/pkg/mod/github.com/spf13/cobra@v1.6.0/command.go:968\nmain.main\n\t/mnt/vss/_work/1/s/gopath/src/github.com/azure-core/ClusterConfigurationAgent/kube-aad-proxy/main.go:32\nruntime.main\n\t/mnt/vss/_work/1/go/src/runtime/proc.go:250\nruntime.goexit\n\t/mnt/vss/_work/1/go/src/runtime/asm_amd64.s:1598"
Interesting thing is that the wildcard domain its alerting on isn't associated with this machine at all. It is within our network, but not even attached to the clusters public ip address. I'm not really sure how it even has that certificate.
-
kobulloc-MSFT 23,491 Reputation points Microsoft Employee
2024-02-06T21:54:06.4666667+00:00 Hello, @Ryan Taylor ! To further troubleshoot this we'll need to take a closer look at your resources. Please email the following to AzCommunity@microsoft.com and we'll get back to you promptly:
- Subject: "Attn: kobulloc - Azure Arc-Enabled Kubernetes kube-aad-proxy"
- Email body: Your Subscription ID
- Email body: A link to this thread so we can validate and expedite the request
If you don't receive a response within 24 hours, please reply to the thread so we can investigate.
-
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(182.40000000000003, 81%, 27%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EMV%3C/text%3E%3C/svg%3E)
Milind Vinayak Belhe 0 Reputation points2024-04-19T03:36:08.2333333+00:00 Hello, I am getting the same issue. Appreciate a response.
Sign in to comment