Failure to create service connection between AKS and KV

Question

Failure to create service connection between AKS and KV

Martin Calsyn 0

Upon following the directions at https://learn.microsoft.com/en-us/azure/aks/csi-secrets-store-identity-access?tabs=azure-portal&pivots=access-with-service-connector, both the CLI and portal gui result in the error shown below. Please assist me in getting past this and creating the connection.

Error: Operation is not succeeded: Failed. {"code":"ExternalServiceError","message":"Execution failed. The extension operation failed with the following error: Error: [ InnerError: [Helm installation failed : : InnerError [release sc-extension failed, and has been uninstalled due to atomic being set: failed post-install: 1 error occurred:\n\t* job sc-job failed: BackoffLimitExceeded\n\n]]] occurred while doing the operation : [Create] on the config . For general troubleshooting visit: https://aka.ms/k8s-extensions-TSG .\nStatus: 200 (OK)\nErrorCode: ExtensionOperationFailed\n\nService request succeeded. Response content and headers are not included to avoid logging sensitive data.\n"}

Martin Calsyn 0 Reputation points

2025-11-21T13:25:57.1533333+00:00

The AI-generated answer was incomplete, but gave me the hint of looking at pods in the sc-system namespace. See the result below. This is an arm64 cluster. It looks like whatever it is trying to start is maybe not arm64 compatible.

mcalsyn@macstudio multitenant % kubectl -n sc-system get pods

NAME READY STATUS RESTARTS AGE

sc-job-54x9p 0/1 Error 0 2m7s

sc-job-f2rth 0/1 Error 0 117s

sc-job-t9q2g 0/1 Error 0 97s

sc-job-v7wj4 0/1 Error 0 57s

mcalsyn@macstudio multitenant % kubectl -n sc-system logs sc-job-v7wj4

exec /main: exec format error

mcalsyn@macstudio multitenant % kubectl -n sc-system logs sc-job-t9q2g

exec /main: exec format error
Manish Deshpande 1,170 Reputation points Microsoft External Staff Moderator

2025-11-21T14:34:01.9966667+00:00
Hello @Martin Calsyn

Your AKS cluster is on arm64, but the Service Connector extension is pulling an amd64-only image, causing an exec format error.

To resolve this, we can either use an arm64-compatible Service Connector image.

skip it and use Workload Identity or managed identity with the CSI driver, which works on arm64.

Firstly we will run the below script to confirm if there is any Mismatch in configuration.

# 1) Prove nodes are arm64 kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.nodeInfo.architecture}{"\n"}{end}' # 2) See the image# 2) See the image that sc-job uses kubectl get job sc-job -n sc-system -o yaml | grep -A2 'image:' # 3) Check whether the image tag includes linux/arm64 # (needs Docker/nerdctl; if not available, just note the tag and consult your registry)

Link - https://learn.microsoft.com/en-us/azure/service-connector/how-to-use-service-connector-in-aks

If the manifest just shows amd64, that's basically your answer right there.

Reinstall/update the Service Connector extension with an arm64-capable image so sc-job starts successfully.

Remove Failed extension.

helm uninstall serviceconnector -n schelm uninstall serviceconnector -n sc az k8s-extension delete \ --cluster-type managedClusters \ --cluster-name <AKS_NAME> \ --resource-group <RG> \

Then purge any leftover Helm release:
helm uninstall serviceconnector -n sc-system || true

Go ahead and reinstall using a tag that covers arm64. If you’ve got any internal docs or a version that puts out multi-arch images, just pin that one instead.
--cluster-type managedClusters \ --cluster-name <AKS_NAME> \ --resource-group <RG> \ --name serviceconnector \ --extension-type Microsoft.Azure.ServiceConnector \ --version <multi-arch-version> \ --release-train stable
b. If your org tracks tested Service Connector builds, just pick one that lists linux/arm64 in its image manifest for sc-job (multi-arch). Once you’ve set it up, keep an eye on the job to see how it runs.
azurepowershell kubectl -n sc-system get job kubectl

The usual troubleshooting steps for Service Connector include looking at the sc-system namespace, checking the Helm release status, and reviewing the sc-job logs. C. Create the connection again (portal or CLI). Follow the “How to use Service Connector in AKS” steps to (re)create the Key Vault connection after the extension is healthy. [[How to use...soft Learn | Learn.Microsoft.com]](https://learn.microsoft.com/en-us/azure/service-connector/how-to-use-service-connector-in-aks)

Second Option : Skip Service Connector; use CSI Driver with Workload Identity (works on arm64)

connect AKS → Key Vault using the CSI Secrets Store provider and Workload Identity/managed identity, which runs fine on arm64.

Link - https://learn.microsoft.com/en-us/azure/aks/csi-secrets-store-identity-access?tabs=azure-portal&pivots=access-with-service-connector

Enable the CSI Secrets Store + Key Vault provider add‑on on your AKS:

Follow the AKS Key Vault CSI driver guidance (Workload Identity / managed identity).

Create/assign the managed identity with appropriate Key Vault RBAC and federate it to your AKS service account via Workload Identity.

Apply secretproviderclass to mount Key Vault secrets/certs into your pods.

Deploy a test pod that consumes the mounted secrets to validate end‑to‑end.

Link - https://learn.microsoft.com/en-us/azure/service-connector/tutorial-python-aks-keyvault-csi-driver?tabs=azure-portal
Clean-up steps (to remove the broken install)

# Remove failing pods/jobs kubectl delete job -n sc-system sc-job --ignore-not-found kubectl delete ns sc-system --ignore-not-found # If Helm release exists: helm list -n sc-system

Then choose Path A (fix Service Connector with multi-arch) or Path B (use CSI + Workload Identity without Service Connector).

If you have any question please revert back we are happy to assist you,
Martin Calsyn 0 Reputation points

2025-11-21T14:53:19.1933333+00:00

working through the suggested steps now...
Martin Calsyn 0 Reputation points

2025-11-21T15:05:16.5533333+00:00

The command 'az k8s-extension delete' is incomplete. Needs the extension name. What is the full extension name? I tried 'serviceconnector' but that was incorrect.
Adam Zachary 2,025 Reputation points

2025-11-21T15:06:49.0566667+00:00

A while ago I ran into almost the exact same issue when helping a team set up a service-connector link between AKS and Key Vault on an ARM64 cluster, and the behavior was identical.

The connector deployment kept failing with the same BackoffLimitExceeded and the pod logs showed exec format error. That error is a classic sign that the extension is pulling AMD64 images and trying to run them on an ARM64 node pool.

When that happens, the Helm chart never completes the post-install job, so the whole Service Connector operation fails. Nothing is wrong with your Key Vault or the identity configuration. It is simply an architecture mismatch.

The only reliable fix was:

Use an AMD64 node pool, even temporarily, because today the Service Connector extension and its CSI helper images are not built for ARM64.

Install the extension again from the AMD64 pool.

Once the connector is deployed and CSI is working, you can decide whether to continue running mixed architecture or migrate workloads accordingly.

If running ARM64 only is a hard requirement, then the answer is that the current extension is not ARM64-compatible and you would need to open a support ticket so Microsoft can confirm image availability for your region.

So, the root cause is not your config. It is simply that the Service Connector / CSI extension does not support pure ARM64 clusters yet.
Martin Calsyn 0 Reputation points

2025-11-21T15:11:50.01+00:00

@Manish Deshpande Ok - first block of commands completed successfully. There are no residual components left behind.

The second block of commands is incomplete - it just starts with arguments and is missing the <multi-arch-version> argument. What should that be.
Manish Deshpande 1,170 Reputation points Microsoft External Staff Moderator

2025-11-21T15:12:36.8+00:00
Hello Martin

List all extensions on the AKS cluster

az k8s-extension list \ --cluster-type managedClusters \ --resource-group <RG_NAME> \ --cluster-name <AKS_NAME> \ -o table

In the output, look for the row where ExtensionType is Microsoft.Azure.ServiceConnector.
Manish Deshpande 1,170 Reputation points Microsoft External Staff Moderator

2025-11-21T15:14:17.7966667+00:00

Hello Martin

I have requested few details over the private chat could you please help me with it.
Martin Calsyn 0 Reputation points

2025-11-21T15:15:01.83+00:00

[deleted]
Manish Deshpande 1,170 Reputation points Microsoft External Staff Moderator

2025-11-25T12:57:13.4+00:00

Hello @Martin Calsyn

I wanted to check if my last response made sense. I’d be glad to assist further or explain anything in more detail and please accept as Yes and upvote if the answer is helpful so that it can help others in the community.
Martin Calsyn 0 Reputation points

2025-11-28T10:28:12+00:00

@Adam Zachary , sorry to be slow to respond to your suggested answer, which would solve the problem by throwing money at it (Intel nodes are more expensive), and there's nothing wrong with that, but I have been pursuing a resolution with MS Support in private messages and email. I will report back here if/when we get a solution.Another issue with switching to Intel nodes is that some Azure credits come with limits on when and how they can be spent, and that also limited my choices a bit. Even through I am now spending my own funds, I want to spend them efficiently, and that favors arm nodes. I'm hoping that by working with msft we can get a proper solution in place so that the documented procedure works across the amd64 and arm64 node types. In the meantime, I will probably use workload identities or k8s textual secrets as a bridge while we await a proper solution.
Shraddha Pandey 390 Reputation points Microsoft External Staff Moderator

2025-12-01T10:36:03.7366667+00:00
Hi Martin Calsyn,

You are experiencing a known Service Connector/AKS integration issue during the Helm deployment stage ("release sc-extension failed... job sc-job failed: BackoffLimitExceeded"). This usually indicates that a post-install Kubernetes Job, which typically handles prerequisite checks or setup tasks, has failed repeatedly or was unable to finish because of problems with your cluster configuration or available resources.

The error “BackoffLimitExceeded” usually points to:

Missing cluster prerequisites (CSI driver, permissions).

Resource limits (insufficient memory/CPU for the Job).

Incomplete/failed registration of resource providers.

Networking or DNS/connectivity issues inside your AKS cluster.

Cluster is missing necessary add-ons or using preview features that aren't enabled.

1.Verify Prerequisites & Provider Registration

Ensure both Microsoft.ServiceLinker and Microsoft.KubernetesConfiguration are registered:

Ensure AKS cluster already has the Azure Key Vault CSI driver enabled.

az provider show -n "Microsoft.ServiceLinker" --query registrationState

az provider show -n "Microsoft.KubernetesConfiguration" --query registrationState

2.Check Cluster Add-ons

az aks show --resource-group <resource-group-name> --name <cluster-name>

Confirm presence of the azureKeyvaultSecretsProvider add-on in the output.

3.Inspect Problematic Job and Logs

Check for failed Jobs in the kube-system or target namespace:

kubectl get jobs -A

kubectl describe job <sc-job-name> -n <namespace>

4.Inspect logs of failed pods

kubectl get pods -A | grep sc-job

kubectl logs <failed-pod-name> -n <namespace>

5.Confirm Sufficient Cluster Resources

Verify node capacity and that cluster nodes are healthy (kubectl get nodes).

Insufficient compute in small test clusters (common in Students subscriptions) can block Jobs from completing.

If there’s tight resource quotas, consider scaling your node pool up or using larger instances temporarily.

6..Extension/Addon Recovery

If stuck in a failed state:

Uninstall failed extension/add-on (if present): az aks disable-addons --addons azure-keyvault-secrets-provider --resource-group <resource-group> --name <cluster-name>

Allow several minutes, then re-enable and retry the Service Connector creation .

Make sure to delete zombie SecretProviderClass resources if present, as their existence can block proper add-on teardown.

Documents:

Troubleshoot errors when deploying AKS cluster extensions

Connect your Azure identity provider to the Azure Key Vault Secrets Store CSI Driver in Azure Kubernetes Service (AKS)

Please let us know if you have any further questions on this.

Thank You!
Shraddha Pandey 390 Reputation points Microsoft External Staff Moderator

2025-12-03T08:20:18.19+00:00

Hi Martin Calsyn,

Just checking to see if you have a chance to check my previous response and helped, do let me know if you have any further questions on this.

Thank You!
Martin Calsyn 0 Reputation points

2025-12-03T15:25:36.8+00:00
I have completed all of the steps above with the following result.

Both show as "Registered"

Present::
mcalsyn@Martins-MacBook-Pro cloud % az aks show --resource-group lucendus-eu-fr --name prometheus | grep -i keyvault "azureKeyvaultSecretsProvider": { "resourceId": "/subscriptions/ee71622c-61bc-4c5d-b3e1-3da221485cd7/resourcegroups/MC_lucendus-eu-fr_prometheus_francecentral/providers/Microsoft.ManagedIdentity/userAssignedIdentities/azurekeyvaultsecretsprovider-prometheus" "azureKeyVaultKms": null,

The 'describe' output shows no errors.

The jobs logs show this error: "exec /main: exec format error" which is the same error that I reported from this same command on Nov 21

There is adequate headroom on all nodes.

The recovery steps just resulted in the same outcome.

The error, IMHO, still points to a platform mismatch (i.e., the job container not being compatible with the arm64 node type).
Shraddha Pandey 390 Reputation points Microsoft External Staff Moderator

2025-12-04T15:16:11.3366667+00:00

Hi Martin Calsyn,

Please check DNS resolution from inside your AKS cluster. You can run the following command to start a test pod and use nslookup to verify if your cluster can resolve the Key Vault domain name:

kubectl run -i --tty dns-check --image=busybox --restart=Never -- sh

nslookup <your-keyvault-name>.vault.azure.net and

If the DNS resolution fails, your cluster won't be able to connect to Azure Key Vault, and any job depending on it will fail.

Also, note that assigning the Key Vault Certificates Officer role to a managed identity allows it to manage certificates, but it is not sufficient for accessing secrets or connecting to Key Vault. Please ensure the identity also has a role like Key Vault Administrator or Key Vault Secrets User, depending on your access needs. azure-built-in-roles-for-key-vault-data-plane-operations.

Please refer the below document for additionally.

https://learn.microsoft.com/en-us/azure/aks/csi-secrets-store-nginx-tls

If you have any further questions or need additional assistance, please feel free to let me know

Thank You!
Martin Calsyn 0 Reputation points

2025-12-04T18:55:56.0566667+00:00

Yes, the nslookup completed successfully and returned correct addresses. This is not a DNS problem.

Regarding the link you sent, it is not relevant. I am not trying to set up TLS via CSI. I have that set up already and it is working.

Please refer to the learn.microsoft.com link in my original posting. Those are the instructions that are failing. I am trying to set up access to non-certificate secrets from pod processes.

Have you repeated those steps successfully yourself on an arm64 Standard_D2ps_v6 node pool? The error indicates an exec failure,, not a lookup failure. This sure sounds like the sc-job container does not have a proper arm64 image in it.
Shraddha Pandey 390 Reputation points Microsoft External Staff Moderator

2025-12-05T14:11:22.7333333+00:00

Hi Martin Calsyn,

Currently, there is no container image available for ARM64 architecture for this solution.

The solution is to switch the nodes to a type with AMD64 architecture.

If you have any further questions or need additional assistance, please feel free to let me know.

Thank You!

1 answer

Your answer

Martin Calsyn 0 Reputation points

2025-11-21T13:25:57.1533333+00:00

The AI-generated answer was incomplete, but gave me the hint of looking at pods in the sc-system namespace. See the result below. This is an arm64 cluster. It looks like whatever it is trying to start is maybe not arm64 compatible.

mcalsyn@macstudio multitenant % kubectl -n sc-system get pods

NAME READY STATUS RESTARTS AGE

sc-job-54x9p 0/1 Error 0 2m7s

sc-job-f2rth 0/1 Error 0 117s

sc-job-t9q2g 0/1 Error 0 97s

sc-job-v7wj4 0/1 Error 0 57s

mcalsyn@macstudio multitenant % kubectl -n sc-system logs sc-job-v7wj4

exec /main: exec format error

mcalsyn@macstudio multitenant % kubectl -n sc-system logs sc-job-t9q2g

exec /main: exec format error
Martin Calsyn 0 Reputation points

2025-11-21T14:53:19.1933333+00:00

working through the suggested steps now...
Martin Calsyn 0 Reputation points

2025-11-21T15:05:16.5533333+00:00

The command 'az k8s-extension delete' is incomplete. Needs the extension name. What is the full extension name? I tried 'serviceconnector' but that was incorrect.
Adam Zachary 2,025 Reputation points

2025-11-21T15:06:49.0566667+00:00

A while ago I ran into almost the exact same issue when helping a team set up a service-connector link between AKS and Key Vault on an ARM64 cluster, and the behavior was identical.

The connector deployment kept failing with the same BackoffLimitExceeded and the pod logs showed exec format error. That error is a classic sign that the extension is pulling AMD64 images and trying to run them on an ARM64 node pool.

When that happens, the Helm chart never completes the post-install job, so the whole Service Connector operation fails. Nothing is wrong with your Key Vault or the identity configuration. It is simply an architecture mismatch.

The only reliable fix was:

Use an AMD64 node pool, even temporarily, because today the Service Connector extension and its CSI helper images are not built for ARM64.

Install the extension again from the AMD64 pool.

Once the connector is deployed and CSI is working, you can decide whether to continue running mixed architecture or migrate workloads accordingly.

If running ARM64 only is a hard requirement, then the answer is that the current extension is not ARM64-compatible and you would need to open a support ticket so Microsoft can confirm image availability for your region.

So, the root cause is not your config. It is simply that the Service Connector / CSI extension does not support pure ARM64 clusters yet.
Martin Calsyn 0 Reputation points

2025-11-21T15:11:50.01+00:00

@Manish Deshpande Ok - first block of commands completed successfully. There are no residual components left behind.

The second block of commands is incomplete - it just starts with arguments and is missing the <multi-arch-version> argument. What should that be.
Manish Deshpande 1,170 Reputation points Microsoft External Staff Moderator

2025-11-21T15:12:36.8+00:00

Hello Martin

List all extensions on the AKS cluster

az k8s-extension list \ --cluster-type managedClusters \ --resource-group <RG_NAME> \ --cluster-name <AKS_NAME> \ -o table

In the output, look for the row where ExtensionType is Microsoft.Azure.ServiceConnector.
Manish Deshpande 1,170 Reputation points Microsoft External Staff Moderator

2025-11-21T15:14:17.7966667+00:00

Hello Martin

I have requested few details over the private chat could you please help me with it.
Martin Calsyn 0 Reputation points

2025-11-21T15:15:01.83+00:00

[deleted]
Manish Deshpande 1,170 Reputation points Microsoft External Staff Moderator

2025-11-25T12:57:13.4+00:00

Hello @Martin Calsyn

I wanted to check if my last response made sense. I’d be glad to assist further or explain anything in more detail and please accept as Yes and upvote if the answer is helpful so that it can help others in the community.
Martin Calsyn 0 Reputation points

2025-11-28T10:28:12+00:00

@Adam Zachary , sorry to be slow to respond to your suggested answer, which would solve the problem by throwing money at it (Intel nodes are more expensive), and there's nothing wrong with that, but I have been pursuing a resolution with MS Support in private messages and email. I will report back here if/when we get a solution.Another issue with switching to Intel nodes is that some Azure credits come with limits on when and how they can be spent, and that also limited my choices a bit. Even through I am now spending my own funds, I want to spend them efficiently, and that favors arm nodes. I'm hoping that by working with msft we can get a proper solution in place so that the documented procedure works across the amd64 and arm64 node types. In the meantime, I will probably use workload identities or k8s textual secrets as a bridge while we await a proper solution.
Shraddha Pandey 390 Reputation points Microsoft External Staff Moderator

2025-12-03T08:20:18.19+00:00

Hi Martin Calsyn,

Just checking to see if you have a chance to check my previous response and helped, do let me know if you have any further questions on this.

Thank You!
Martin Calsyn 0 Reputation points

2025-12-03T15:25:36.8+00:00

I have completed all of the steps above with the following result.

Both show as "Registered"

Present::
mcalsyn@Martins-MacBook-Pro cloud % az aks show --resource-group lucendus-eu-fr --name prometheus | grep -i keyvault "azureKeyvaultSecretsProvider": { "resourceId": "/subscriptions/ee71622c-61bc-4c5d-b3e1-3da221485cd7/resourcegroups/MC_lucendus-eu-fr_prometheus_francecentral/providers/Microsoft.ManagedIdentity/userAssignedIdentities/azurekeyvaultsecretsprovider-prometheus" "azureKeyVaultKms": null,

The 'describe' output shows no errors.

The jobs logs show this error: "exec /main: exec format error" which is the same error that I reported from this same command on Nov 21

There is adequate headroom on all nodes.

The recovery steps just resulted in the same outcome.

The error, IMHO, still points to a platform mismatch (i.e., the job container not being compatible with the arm64 node type).
Shraddha Pandey 390 Reputation points Microsoft External Staff Moderator

2025-12-04T15:16:11.3366667+00:00

Hi Martin Calsyn,

Please check DNS resolution from inside your AKS cluster. You can run the following command to start a test pod and use nslookup to verify if your cluster can resolve the Key Vault domain name:

kubectl run -i --tty dns-check --image=busybox --restart=Never -- sh

nslookup <your-keyvault-name>.vault.azure.net and

If the DNS resolution fails, your cluster won't be able to connect to Azure Key Vault, and any job depending on it will fail.

Also, note that assigning the Key Vault Certificates Officer role to a managed identity allows it to manage certificates, but it is not sufficient for accessing secrets or connecting to Key Vault. Please ensure the identity also has a role like Key Vault Administrator or Key Vault Secrets User, depending on your access needs. azure-built-in-roles-for-key-vault-data-plane-operations.

Please refer the below document for additionally.

https://learn.microsoft.com/en-us/azure/aks/csi-secrets-store-nginx-tls

If you have any further questions or need additional assistance, please feel free to let me know

Thank You!
Martin Calsyn 0 Reputation points

2025-12-04T18:55:56.0566667+00:00

Yes, the nslookup completed successfully and returned correct addresses. This is not a DNS problem.

Regarding the link you sent, it is not relevant. I am not trying to set up TLS via CSI. I have that set up already and it is working.

Please refer to the learn.microsoft.com link in my original posting. Those are the instructions that are failing. I am trying to set up access to non-certificate secrets from pod processes.

Have you repeated those steps successfully yourself on an arm64 Standard_D2ps_v6 node pool? The error indicates an exec failure,, not a lookup failure. This sure sounds like the sc-job container does not have a proper arm64 image in it.
Shraddha Pandey 390 Reputation points Microsoft External Staff Moderator

2025-12-05T14:11:22.7333333+00:00

Hi Martin Calsyn,

Currently, there is no container image available for ARM64 architecture for this solution.

The solution is to switch the nodes to a type with AMD64 architecture.

If you have any further questions or need additional assistance, please feel free to let me know.

Thank You!

Answer 1

So the answer (alluded to earlier, and confirmed in the 5-Dec comment by msft above) seems to be that this feature is not supported across the full set of Azure k8s nodes. That should be made clear in the learn.microsoft.com documentation to save us all a bit of time.

It's disappointing that we are releasing Azure features like this that aren't available on the most cost and energy efficient node architectures, especially given the relatively simple fix of cross compiling for those architectures.

So the solution here is "spend more", though I think another alternative that hasn't been mentioned is to use workload managed identities and associated RBAC to access the key vault.

Share via

Failure to create service connection between AKS and KV

1 answer

Your answer