Azure AKS Extension works for me, but not for EU customer.

Question

Azure AKS Extension works for me, but not for EU customer.

Trevor Blackman 20

My company has released an AKS Extension on the marketplace. It deploys successfully for us every time, but for one of our customers in EU West, it hangs for ~20m, and then fails. The error message is extremely generic: "Failed to deploy." I have created an AKS cluster in the EU West region, and it still deploys for me first try every time.

What additional information can I collect to help identify this issue?

Anonymous

2025-08-05T06:42:05.52+00:00
Hello Trevor Blackman,

Thank you for posting your question on Microsoft Q&A.

Adding few more additional details to the answer provided by Nikhil Duserla,

Since the error message is generic ("Failed to deploy"), here are few steps to resolve the issue:

1. Ensure the AKS Cluster Uses Managed Identity

AKS extensions require managed identity. If the customer's cluster is using a service principle, extension deployment will fail. So, update your existing cluster to a user-managed identity by running the below command in PowerShell:

Update-AzAksCluster -ResourceGroupName "<rg-name>" -Name "<cluster-name>" -EnableManagedIdentity

Check the link here :Use a managed identity in Azure Kubernetes Service (AKS) - Azure Kubernetes Service | Microsoft Learn

2. Check Azure Activity Logs

Check for any failed deployment logs or policy blocks using Azure Monitor by running this command in PowerShell:

Get-AzActivityLog -ResourceGroup "<rg-name>" -StartTime (Get-Date).AddDays(-1) | Where-Object {$_.OperationName -like "kubernetesconfiguration/extensions"}

3. Inspect your AKS Extension and Kubernetes Events by running these commands:

Get-AzAksExtension -ResourceGroupName "<rg-name>" -ClusterName "<cluster-name>" -ClusterType "managedClusters"

kubectl get events -A( Kubernetes events)

4. Check for Network or Policy Restrictions

Check whether:

Azure Policies are blocking extensions .

NSGs, firewalls, or Private Link setups are blocking required outbound traffic.

Required Microsoft service FQDNs or IPs are accessible.

5. Review Extension-Related Logs in Kubernetes

If your extension deploys workloads (e.g., via Helm), inspect the deployed pod logs using the below Bash commands:

kubectl get pods -n <extension-namespace>

kubectl logs <pod-name> -n <extension-namespace>

Please follow the above steps to troubleshoot the issue. If the issue still persists provide few more details to help you further identify the issue:

Extension name and version,Whether the AKS cluster is private or public,Output from Activity Logs or kubectl logs

Please let me know if you need any further clarification or assistance.

Thanks,

Rashmika
Anonymous

2025-08-07T11:59:19.4866667+00:00

Hi Trevor Blackman,

Just following up to confirm if you were able to identify the issue using the above steps. Please let me know if you need any further clarification or assistance.

Thanks,

Rashmika
Anonymous

2025-08-12T12:33:40.28+00:00

Hi Trevor Blackman,

Just following up to confirm if you were able to identify the issue using the above steps. Please let me know if you need any further clarification or assistance.

Thanks,

Rashmika
Trevor Blackman 20 Reputation points

2025-08-12T14:41:39.1766667+00:00

Hi Rashmika,

Thank you for your suggestions. I am still testing each of the solutions, an I hope to provide an update by the end of this week.
Anonymous

2025-08-13T07:25:23.8966667+00:00

Hi Trevor Blackman,

Thanks for the information. Let me know if you require further assistance or help from our end.
Anonymous

2025-08-18T07:40:12.41+00:00

Hi Trevor Blackman,

Just following up to confirm if you were able to test the solutions provided. Please let me know if you need any further clarification or assistance.

Thanks,

Rashmika
Anonymous

2025-08-20T12:42:40.6133333+00:00

Hi Trevor Blackman,

Just following up to confirm if you were able to test the solutions provided. Please let me know if you need any further clarification or assistance.

Thanks,

Rashmika

Trevor Blackman 20

I did complete further testing.

We are using a managed AKS cluster, so that is not the issue.

We used this this command and got some output: Get-AzActivityLog -ResourceGroup "<rg-name>" -StartTime (Get-Date).AddDays(-1) | Where-Object {$_.OperationName -like "*kubernetesconfiguration/extensions*"}

I attached the complete output, but here is the line that I think shows the error:

                       statusMessage  : {"status":"Failed","error":{"code":"ResourceOperationFailure","message":"The resource operation completed with terminal provisioning state 'Failed'.","details":[{"code"
                       :"ExtensionOperationFailed","message":"The extension operation failed with the following error:  Error: [ InnerError: [Helm installation failed : Unable to render the Helm chart with th
                       e provided config settings and config protected settings : Recommendation Please check if the values provided to the config settings and the config protected settings are valid for this
                        extension type : InnerError [YAML parse error on mte-api-relay/templates/deployment.yaml: error converting YAML to JSON: yaml: line 27: did not find expected key]]] occurred while doin
                       g the operation : [Create] on the config, For general troubleshooting visit: https://aka.ms/k8s-extensions-TSG.","additionalInfo":[]}]}}
                       eventCategory  : Administrative
                       entity         : /subscriptions/350d9868-0736-4ea7-b8f7-a52fb7a9029d/resourcegroups/eclypses/providers/Microsoft.ContainerService/managedClusters/eclypsesmteapi/providers/Microsoft.Kube
                       rnetesConfiguration/extensions/mteapirelay
                       message        : Microsoft.KubernetesConfiguration/extensions/write
                       hierarchy      : 2fc16a66-9323-4c27-b34a-0e0549438228/350d9868-0736-4ea7-b8f7-a52fb7a9029d

Specifically,

error converting YAML to JSON: yaml: line 27: did not find expected key

When I look at the YAML file, line 27 is where I'm mapping user input to environment variables inside the kubernetes extensions. The specific line is name: APP_SETTINGS__OUTBOUND_TOKEN

          env:
            - name: APP_SETTINGS__UPSTREAM
              value: "{{ .Values.upstream }}"
            - name: APP_SETTINGS__CLIENT_ID_SECRET
              value: "{{ .Values.clientIdSecret }}"
            - name: APP_SETTINGS__OUTBOUND_TOKEN
              value: "{{ .Values.outboundToken }}"
            - name: APP_SETTINGS__SECRET
              value: "{{ .Values.secret }}"

Launching this AKS extension fails with this error 100% of the time for our customer, but when I test on my own account it works 100% of the time. I also opened a completely new personal account, and the extension deploys perfectly there as well, every time.

What do you think could be happening, and what additional trouble shooting steps can I take?

Answer accepted by question author

2 additional answers

Your answer

Anonymous

2025-08-07T11:59:19.4866667+00:00

Hi Trevor Blackman,

Just following up to confirm if you were able to identify the issue using the above steps. Please let me know if you need any further clarification or assistance.

Thanks,

Rashmika
Anonymous

2025-08-12T12:33:40.28+00:00

Hi Trevor Blackman,

Just following up to confirm if you were able to identify the issue using the above steps. Please let me know if you need any further clarification or assistance.

Thanks,

Rashmika
Trevor Blackman 20 Reputation points

2025-08-12T14:41:39.1766667+00:00

Hi Rashmika,

Thank you for your suggestions. I am still testing each of the solutions, an I hope to provide an update by the end of this week.
Anonymous

2025-08-13T07:25:23.8966667+00:00

Hi Trevor Blackman,

Thanks for the information. Let me know if you require further assistance or help from our end.
Anonymous

2025-08-18T07:40:12.41+00:00

Hi Trevor Blackman,

Just following up to confirm if you were able to test the solutions provided. Please let me know if you need any further clarification or assistance.

Thanks,

Rashmika
Anonymous

2025-08-20T12:42:40.6133333+00:00

Hi Trevor Blackman,

Just following up to confirm if you were able to test the solutions provided. Please let me know if you need any further clarification or assistance.

Thanks,

Rashmika

Answer 1

Hello Trevor Blackman,

Thank you for following steps suggested and providing us with the logs. Based on the error

Helm installation failed: YAML parse error on mte-api-relay/templates/deployment.yaml:

error converting YAML to JSON: yaml: line 27: did not find expected key

The issue is not related to managed identity or network connectivity, but rather to how the Helm chart is being rendered with the values provided during extension deployment. Specifically, the value being passed to outboundToken is not being parsed correctly by Helm, which leads to the failure.

Here are the next steps to troubleshoot the error based on the logs provided:

1.Provide the Exact Values Used: exact command you have used to deploy the extension-configuration settings and config-protected-settings values (you can remove any secrets) to verify what value is being passed for outboundToken, since special characters, empty values, or multiline inputs can cause YAML parsing errors.

You need to modify the command to ensure the outboundToken value is a correctly formatted YAML string. The simplest way to do this is to enclose the value in double quotes (").

__Example:__If the previous command was like this:

Bash

az k8s-extension create \

--cluster-type managedClusters \

--cluster-name <cluster-name> \

--resource-group <resource-group> \

--name mteapirelay \

--extension-type "MTEApiRelayExtension" \

--scope cluster \

--configuration-settings '{"outboundToken": some-token-value}'

It should be changed to this: Bash

az k8s-extension create \

--cluster-type managedClusters \

--cluster-name <cluster-name> \

--resource-group <resource-group> \

--name mteapirelay \

--extension-type "MTEApiRelayExtension" \

--scope cluster \

--configuration-settings '{"outboundToken": "some-token-value"}'

By adding the double quotes around the token value, it is correctly parsed as a string, which should prevent the Helm chart from failing.

For more information, refer here: Helm | Template Functions and Pipelines

After correcting the command, the customer should attempt to re-install the AKS extension. The new command should successfully parse the YAML, and the extension should deploy correctly.

For more information, refer here: Cluster extensions for Azure Kubernetes Service (AKS) - Azure Kubernetes Service | Microsoft Learn

2.Test Rendering Locally: With the same values, try rendering the Helm chart locally. If the error reproduces, it confirms the issue is with the provided input rather than the cluster.

helm template my-release ./<chart-path> -f values.yaml

For more information, refer here: Helm | Helm Template

3.Inspect the values applied in the cluster: After attempting deployment, check how the environment variables were rendered in the Deployment manifest. This will whether the value was truncated, malformed, or missing.

kubectl get deployment <extension-deployment> -n <extension-namespace> -o yaml | grep APP_SETTINGS__OUTBOUND_TOKEN -A2

4.Validate Cluster Context: Provide the AKS version __.__Once confirm whether the cluster is public or private. Any Azure Policies or admission controllers (OPA/Gatekeeper) applied that could mutate resource manifests.

__AKS version (az aks show --resource-group <rg> --name <cluster> --query kubernetesVersion)- __check the version using this

For more information, refer here: Built-in policy definitions for Azure Kubernetes Service - Azure Kubernetes Service | Microsoft Learn

The deployment failure is caused by a YAML parsing error in the Helm chart due to the outboundToken value. Kindly follow the above steps to share the details required:

Share the exact --configuration-settings used.
Try helm template locally with those values.
Confirm AKS version, cluster type (public/private), and any applied Azure Policies.

With this info, we can identify whether the issue is input-related (mostly seems like) or environment-specific. Let me know if you require any additional information from my end. I happy to help you with the queries.

Thanks,

Rashmika

Anonymous

2025-08-28T15:17:28.5133333+00:00

Hello Trevor Blackman,

Just following up to check whether you were able to check the steps provided. Let me know if you require any additional information from my end. I am happy to help you with the queries. If the information is helpful, please click on Upvote and Accept Answer on it so that it can help other community members.

Thanks,

Rashmika

Answer 2

Trevor Blackman 20

This has been resolved.

The issue was the user was putting in a double quote without escaping it, which obviously caused an error with Helm chart parsing.

Thank you for all your support.

Anonymous

2025-08-29T08:11:45.5566667+00:00

Hi Trevor Blackman,

Thanks for the confirmation. Let me know if you require any additional information from my end. I am happy to help you with the queries. If the answer provided by me was helpful, please click on Upvote and Accept Answer on it so that it can help other community members.

Thanks,

Rashmika
Anonymous

2025-08-31T04:21:09.75+00:00

Hi Trevor Blackman,

Thanks for the confirmation. Let me know if you require any additional information from my end. I am happy to help you with the queries. If the answer provided by me was helpful, please click on Upvote and Accept Answer on it so that it can help other community members.

Thanks,

Rashmika

Answer 3

@Trevor Blackman

Ensure that your AKS cluster is created with a managed identity, as cluster extensions won't work with service principal-based clusters.

For new clusters created with az aks create, managed identity is configured by default. For existing service principal-based clusters that need to be switched over to managed identity, it can be enabled by running az aks update with the --enable-managed-identity flag. For more information, see Use managed identity.Troubleshoot errors when deploying AKS cluster extensions: https://learn.microsoft.com/en-us/troubleshoot/azure/azure-kubernetes/extensions/cluster-extension-deployment-errors?source=recommendationsIf you have any further queries, do let us know.

Share via

Azure AKS Extension works for me, but not for EU customer.

2 additional answers

Your answer