Create a service connection in AKS with Service Connector - Key Vault - BackoffLimitExceeded

Question

Create a service connection in AKS with Service Connector - Key Vault - BackoffLimitExceeded

KlemenVezjak-6511 0

Hi,

Im trying to configure Azure Key Vault Secrets Store CSI Driver in AKS. I went through the process described here: https://learn.microsoft.com/en-us/azure/aks/csi-secrets-store-identity-access?tabs=azure-cli&pivots=access-with-service-connector

I get the error:

Error: Operation is not succeeded: Failed. {"code":"ExternalServiceError","message":"Execution failed. The extension operation failed with the following error: Error: [ InnerError: [Helm installation failed : : InnerError [release sc-extension failed, and has been uninstalled due to atomic being set: failed post-install: 1 error occurred:\n\t* job sc-job failed: BackoffLimitExceeded\n\n]]] occurred while doing the operation : [Create] on the config, For general troubleshooting visit: https://aka.ms/k8s-extensions-TSG.\nStatus: 200 (OK)\nErrorCode: ExtensionOperationFailed\n\nService request succeeded. Response content and headers are not included to avoid logging sensitive data.\n"}

I checked problems/solutions here: https://learn.microsoft.com/en-us/troubleshoot/azure/azure-kubernetes/extensions/cluster-extension-deployment-errors#helm-errors

So,

I dont have taints "NoSchedule" on my node pools
Private endpoint of Key Vault and AKS nodes are in the same virtual networks
All "*aks-secrets-store-csi-driver-**" pods are healthy and they are located on each node
"Microsoft.ServiceLinker" and "Microsoft.KubernetesConfiguration" are registered on the target subscription
Managed identity is created and has a Key Vault Certificates Officer RBAC role assigned on target Key Vault (this is deployed with PS command "az aks connection create keyvault --connection...")
I dont have any FW or Virtual Network Appliance to deny or control outbound network traffic

Thank you!

Siva Pavuluri 490 Reputation points Microsoft External Staff Moderator

2025-05-16T12:28:19.6533333+00:00

Hi KlemenVezjak-6511,

I understand you're trying to configure the Azure Key Vault Secrets Store CSI Driver in AKS and are encountering a BackoffLimitExceeded error during the Helm installation process.

To help resolve this, please verify the following:

Ensure the SecretProviderClass configuration is correct—specifically, check that the Key Vault name, tenantId, and objectId are accurately set.

Confirm that the Azure Key Vault Secrets Provider add-on is enabled in your AKS cluster add-on enable.

Additionally, if a large number of pods attempt to access the same Key Vault within a short time frame, you may encounter throttling issues. The CSI driver currently lacks built-in throttling protection, which can lead to failures during initialization. To mitigate this, consider temporarily scaling down the number of pods accessing the Key Vault.

To further investigate, please share the logs from the failed job using the following commands:

kubectl get jobs -n kube-system

kubectl describe job <job-name> -n kube-system

kubectl logs job/<job-name> -n kube-system

Please refer the document for the troubleshoot-key-vault-csi-secrets-store-csi-driver:

https://learn.microsoft.com/en-us/troubleshoot/azure/azure-kubernetes/extensions/troubleshoot-key-vault-csi-secrets-store-csi-driver#step-1-confirm-that-azure-key-vault-secrets-provider-add-on-is-enabled-on-your-cluster

If you have any further questions or need additional assistance, please feel free to let me know. If the information is helpful, please click on Upvote👍.

Thank you.
KlemenVezjak-6511 0 Reputation points

2025-05-19T05:06:43.45+00:00

Hi @Siva Pavuluri ,

thank you for your response.

One of the attempts (out of desperation) was also clicking on the portal. In manual configuration, you select subscription and Key Vault, so the data should be correct.

I also already check provider add-on installation (with command "az aks show --resource-group <your-resource-group> --name <your-aks-name> --query addonProfiles.azureKeyvaultSecretsProvider.enabled" and manually on the portal).

The cluster is empty - there are only system nodes and nginx ingress controller - the plan is to get certificates for TLS/SSL from Key Vault.

Have a nice day,K
Siva Pavuluri 490 Reputation points Microsoft External Staff Moderator

2025-05-19T10:00:40.2633333+00:00

Hi KlemenVezjak-6511,

Please check DNS resolution from inside your AKS cluster. You can run the following command to start a test pod and use nslookup to verify if your cluster can resolve the Key Vault domain name:

kubectl run -i --tty dns-check --image=busybox --restart=Never -- sh

nslookup <your-keyvault-name>.vault.azure.net
and

If the DNS resolution fails, your cluster won't be able to connect to Azure Key Vault, and any job depending on it will fail.

Also, note that assigning the Key Vault Certificates Officer role to a managed identity allows it to manage certificates, but it is not sufficient for accessing secrets or connecting to Key Vault. Please ensure the identity also has a role like Key Vault Administrator or Key Vault Secrets User, depending on your access needs. azure-built-in-roles-for-key-vault-data-plane-operations.

Please refer the below document for additionally.

https://learn.microsoft.com/en-us/azure/aks/csi-secrets-store-nginx-tls

If you have any further questions or need additional assistance, please feel free to let me know.

Thank you.
KlemenVezjak-6511 0 Reputation points

2025-05-19T11:14:37.1233333+00:00

Hi @Siva Pavuluri ,

thank you.

I already assign additional role "Key Vault Secret User".

I already check DNS configuration and it resolve my private IP of the Key Vault (I configured private endpoint). But I noticed something else - during the creation of Service Connector for my Key Vault, I can't select option "Use Private Link to access target service with a private endpoint." under "Network" section - is grayed out. Is that normal? I check the same in one different AKS and it the same.

My AKS nodes and a Key Vault are in the same virtual network, but different subnets. I dont have any NSG network rule for deny traffic between the subnets. In Key Vault FW I allowed traffic from my entire virtual network (I included all of the subnets) and I allowed traffic from my NAT GW, which is configured for my virtual network. I can see that RBAC roles for AKS identity are added, but it fails later, with some configuration or something like that.

Have a nice day,
K
Siva Pavuluri 490 Reputation points Microsoft External Staff Moderator

2025-05-19T12:14:51.3833333+00:00

Hi KlemenVezjak-6511,

The Service Connector for AKS supports only the Firewall Rules networking option. In contrast, other compute services may also support Private Link and Virtual Network options.

Please ensure that the Service Connection authentication is properly configured. If you're encountering issues while creating the Service Connection, kindly share the error message or deployment logs to help troubleshoot the problem more effectively.

Please refer to the document below for a better understanding of AKS integration with Service Connector.

https://learn.microsoft.com/en-us/azure/service-connector/how-to-use-service-connector-in-aks

https://learn.microsoft.com/en-us/azure/service-connector/tutorial-python-aks-storage-workload-identity?tabs=azure-portal#create-service-connection-with-service-connector

If you have any further questions or need additional assistance, please feel free to let me know.

Thank you.
KlemenVezjak-6511 0 Reputation points

2025-05-20T05:10:46.2533333+00:00

Hi @Siva Pavuluri ,

In awareness of this, a NAT gateway was also added to the FW rules, which is configured on the subnet in which the AKS nodes are located.

Entire error is in the first post.

Br., K

1 answer

Your answer

Siva Pavuluri 490 Reputation points Microsoft External Staff Moderator

2025-05-16T12:28:19.6533333+00:00

Hi KlemenVezjak-6511,

I understand you're trying to configure the Azure Key Vault Secrets Store CSI Driver in AKS and are encountering a BackoffLimitExceeded error during the Helm installation process.

To help resolve this, please verify the following:

Ensure the SecretProviderClass configuration is correct—specifically, check that the Key Vault name, tenantId, and objectId are accurately set.

Confirm that the Azure Key Vault Secrets Provider add-on is enabled in your AKS cluster add-on enable.

Additionally, if a large number of pods attempt to access the same Key Vault within a short time frame, you may encounter throttling issues. The CSI driver currently lacks built-in throttling protection, which can lead to failures during initialization. To mitigate this, consider temporarily scaling down the number of pods accessing the Key Vault.

To further investigate, please share the logs from the failed job using the following commands:

kubectl get jobs -n kube-system

kubectl describe job <job-name> -n kube-system

kubectl logs job/<job-name> -n kube-system

Please refer the document for the troubleshoot-key-vault-csi-secrets-store-csi-driver:

https://learn.microsoft.com/en-us/troubleshoot/azure/azure-kubernetes/extensions/troubleshoot-key-vault-csi-secrets-store-csi-driver#step-1-confirm-that-azure-key-vault-secrets-provider-add-on-is-enabled-on-your-cluster

If you have any further questions or need additional assistance, please feel free to let me know. If the information is helpful, please click on Upvote👍.

Thank you.
KlemenVezjak-6511 0 Reputation points

2025-05-19T05:06:43.45+00:00

Hi @Siva Pavuluri ,

thank you for your response.

One of the attempts (out of desperation) was also clicking on the portal. In manual configuration, you select subscription and Key Vault, so the data should be correct.

I also already check provider add-on installation (with command "az aks show --resource-group <your-resource-group> --name <your-aks-name> --query addonProfiles.azureKeyvaultSecretsProvider.enabled" and manually on the portal).

The cluster is empty - there are only system nodes and nginx ingress controller - the plan is to get certificates for TLS/SSL from Key Vault.

Have a nice day,K
Siva Pavuluri 490 Reputation points Microsoft External Staff Moderator

2025-05-19T10:00:40.2633333+00:00

Hi KlemenVezjak-6511,

Please check DNS resolution from inside your AKS cluster. You can run the following command to start a test pod and use nslookup to verify if your cluster can resolve the Key Vault domain name:

kubectl run -i --tty dns-check --image=busybox --restart=Never -- sh

nslookup <your-keyvault-name>.vault.azure.net
and

If the DNS resolution fails, your cluster won't be able to connect to Azure Key Vault, and any job depending on it will fail.

Also, note that assigning the Key Vault Certificates Officer role to a managed identity allows it to manage certificates, but it is not sufficient for accessing secrets or connecting to Key Vault. Please ensure the identity also has a role like Key Vault Administrator or Key Vault Secrets User, depending on your access needs. azure-built-in-roles-for-key-vault-data-plane-operations.

Please refer the below document for additionally.

https://learn.microsoft.com/en-us/azure/aks/csi-secrets-store-nginx-tls

If you have any further questions or need additional assistance, please feel free to let me know.

Thank you.
KlemenVezjak-6511 0 Reputation points

2025-05-19T11:14:37.1233333+00:00

Hi @Siva Pavuluri ,

thank you.

I already assign additional role "Key Vault Secret User".

I already check DNS configuration and it resolve my private IP of the Key Vault (I configured private endpoint). But I noticed something else - during the creation of Service Connector for my Key Vault, I can't select option "Use Private Link to access target service with a private endpoint." under "Network" section - is grayed out. Is that normal? I check the same in one different AKS and it the same.

My AKS nodes and a Key Vault are in the same virtual network, but different subnets. I dont have any NSG network rule for deny traffic between the subnets. In Key Vault FW I allowed traffic from my entire virtual network (I included all of the subnets) and I allowed traffic from my NAT GW, which is configured for my virtual network. I can see that RBAC roles for AKS identity are added, but it fails later, with some configuration or something like that.

Have a nice day,
K
Siva Pavuluri 490 Reputation points Microsoft External Staff Moderator

2025-05-19T12:14:51.3833333+00:00

Hi KlemenVezjak-6511,

The Service Connector for AKS supports only the Firewall Rules networking option. In contrast, other compute services may also support Private Link and Virtual Network options.

Please ensure that the Service Connection authentication is properly configured. If you're encountering issues while creating the Service Connection, kindly share the error message or deployment logs to help troubleshoot the problem more effectively.

Please refer to the document below for a better understanding of AKS integration with Service Connector.

https://learn.microsoft.com/en-us/azure/service-connector/how-to-use-service-connector-in-aks

https://learn.microsoft.com/en-us/azure/service-connector/tutorial-python-aks-storage-workload-identity?tabs=azure-portal#create-service-connection-with-service-connector

If you have any further questions or need additional assistance, please feel free to let me know.

Thank you.
KlemenVezjak-6511 0 Reputation points

2025-05-20T05:10:46.2533333+00:00

Hi @Siva Pavuluri ,

In awareness of this, a NAT gateway was also added to the FW rules, which is configured on the subnet in which the AKS nodes are located.

Entire error is in the first post.

Br., K

Answer 1

KlemenVezjak-6511 0

The problem was the architecture of the chosen size for the nodes. Some VM sizes have architecture ARM64, others AMD64.

There is no container image (for this solution) for ARM64 architecture - It looks like Microsoft fell asleep on this one.

The solution was to replace the nodes - to a type that has AMD64 architecture.

I found a problem with Kubernetes comand: kubectl -n sc-system logs -f sc-job-dvx2p
The response was: exec /main: exec format error

I knew from this response that there was something deeper.

Siva Pavuluri 490 Reputation points Microsoft External Staff Moderator

2025-05-20T10:34:46.79+00:00

Hi KlemenVezjak-6511,

I'm glad the issue is resolved for you finally. Thanks again for sharing the root cause here. Have a good day!

Thank you.

Share via

Create a service connection in AKS with Service Connector - Key Vault - BackoffLimitExceeded

1 answer

Your answer