Production AKS cluster is stuck in Failed provisioningState. Cluster is running unsupported Kubernetes version 1.22.x. All nodepool operations (add/delete/scale) fail with InvalidGalleryImageRef and InternalServerError. Control plane upgrade is blocked du

Question

Production AKS cluster is stuck in Failed provisioningState. Cluster is running unsupported Kubernetes version 1.22.x. All nodepool operations (add/delete/scale) fail with InvalidGalleryImageRef and InternalServerError. Control plane upgrade is blocked du

ITOPS LLM 0

Production AKS cluster is stuck in Failed provisioningState.

Cluster is running unsupported Kubernetes version 1.22.x.

All nodepool operations (add/delete/scale) fail with InvalidGalleryImageRef and InternalServerError.

Control plane upgrade is blocked due to failed state.

Request backend repair and forced upgrade to supported version (1.32.x).

Jilakara Hemalatha 9,255 Reputation points Microsoft External Staff Moderator

2026-01-27T07:19:49.3966667+00:00
Hi ITOPS,

Thank you for sharing the details about your AKS cluster. Based on the information provided, your cluster is currently in a Failed provisioning state and is running an unsupported Kubernetes version (1.22.x). As a result, node pool operations (add, delete, scale) are failing with errors such as InvalidGalleryImageRef and InternalServerError.

While a backend repair is typically required for clusters in this state, we can perform the following investigation steps to better understand the cluster’s condition and plan a recovery path:

Recommended Investigation Steps:

Check Cluster Provisioning State:

az aks show --resource-group <rg> --name <cluster> --query provisioningState

This will confirm the current provisioning state and identify any recent failed operations.

List Node Pools and Their Status:

az aks nodepool list --resource-group <rg> --cluster-name <cluster> --output table

This will show which node pools are affected and their provisioning status.

3.Review Cluster Events and Health:

kubectl get nodes kubectl get pods --all-namespaces kubectl describe node <node-name>

Review any node-level or pod-level failures to gather additional clues.

4.Collect Error Details for Analysis:

Record timestamps of failed operations and Note any InvalidGalleryImageRef or InternalServerError messages.

az aks collect-logs --resource-group <rg> --name <cluster> --file-path aks_logs.zip

Check the activity logs in the Azure portal to get more insights about what might be causing the issue. Filter events by Status: Failed and review the error messages.

5.Address Gallery Image Issues (If Applicable):

Identify any images in delete_failed state:

get-mocgalleryimage -location <MocLocation> | Where-Object { $_.properties.statuses.ProvisionState -match "delete_failed" }

Please follow the steps above and share the outputs. This will help us determine the best path to restore your cluster to a supported and healthy state.

Additionally, could you please check the private message and provide necessary information.
Jilakara Hemalatha 9,255 Reputation points Microsoft External Staff Moderator

2026-01-28T07:19:36.11+00:00

Hi ITOPS LLM,

Could you please check the private message and provide necessary information for further investigation.
Himanshu Shekhar 3,465 Reputation points Microsoft External Staff Moderator

2026-01-29T07:08:15.8666667+00:00

Hi ITOPS LLM,

Could you please check the private message and provide necessary information for further investigation.
Jilakara Hemalatha 9,255 Reputation points Microsoft External Staff Moderator

2026-02-02T09:44:30.33+00:00

Hi ITOPS LLM,

Could you please provide update on this issue. If issue still persists, Could you please check the private message and provide necessary information for further investigation.

1 answer

Your answer

Jilakara Hemalatha 9,255 Reputation points Microsoft External Staff Moderator

2026-01-28T07:19:36.11+00:00

Hi ITOPS LLM,

Could you please check the private message and provide necessary information for further investigation.
Himanshu Shekhar 3,465 Reputation points Microsoft External Staff Moderator

2026-01-29T07:08:15.8666667+00:00

Hi ITOPS LLM,

Could you please check the private message and provide necessary information for further investigation.
Jilakara Hemalatha 9,255 Reputation points Microsoft External Staff Moderator

2026-02-02T09:44:30.33+00:00

Hi ITOPS LLM,

Could you please provide update on this issue. If issue still persists, Could you please check the private message and provide necessary information for further investigation.

Answer 1

To address the issue of your AKS cluster being stuck in a Failed provisioning state, you can follow these troubleshooting steps:

Check the Activity Log: Use the Azure portal or Azure CLI to view the activity log for your cluster. This log can provide insights into the errors causing the failure. Look specifically for error codes and messages related to the InvalidGalleryImageRef and InternalServerError.
Diagnose Node Pool Issues: Since all node pool operations are failing, check the status of the node pool using the az aks nodepool show command. If the provisioning state is Failed, the output will include error messages that can help identify the problem.
Upgrade Control Plane: Since your cluster is running an unsupported Kubernetes version (1.22.x), you may need to force an upgrade to a supported version (1.32.x). This can sometimes resolve provisioning issues. However, ensure that you first resolve any existing errors that are blocking the upgrade.
Repair Backend: If the cluster remains in a failed state, consider requesting a backend repair through Azure support. This can help address underlying issues that are not resolvable through standard troubleshooting steps.
Review Quotas and Policies: Ensure that your subscription has sufficient quota and that there are no policies or resource locks preventing the operations you are trying to perform.

By following these steps, you should be able to identify and resolve the issues causing your AKS cluster to be in a failed state.

References:

Troubleshoot Azure Kubernetes Service clusters or nodes in a failed state

Share via

Production AKS cluster is stuck in Failed provisioningState. Cluster is running unsupported Kubernetes version 1.22.x. All nodepool operations (add/delete/scale) fail with InvalidGalleryImageRef and InternalServerError. Control plane upgrade is blocked du

1 answer

Your answer