Container Apps Job Maximum Allowed Cores exceeded

Question

Container Apps Job Maximum Allowed Cores exceeded

Phillip S. Stenger Jr 20

I have an event driven Container Apps Job which has been running well for about a month. It is using a General Purpose D-series workload profile with 4 vCPU and 16GB memory. The profile is used only by this job as is currently set to a min of 0 instances and a max of 1 instance. The job runs on .5 CPU, 1GB memory and has a max of 8 replicas. Recently we have started noticing frequent issues with running even a single job. The system logs show messages like this:

Maximum Allowed Cores exceeded for the Managed Environment. Please check https://learn.microsoft.com/en-us/azure/container-apps/quotas for resource limits

and

0/4 nodes are available: 1 node(s) had untolerated taint {virtual-kubelet.io/provider: legion}, 3 node(s) didn't match Pod's node affinity/selector. preemption: 0/4 nodes are available: 4 Preemption is not helpful for scheduling.

However, when I look at my usage quotas, we are nowhere near any of them:

{                                                                  
  "value": [                  
    {
      "currentValue": 0,
      "limit": 200,
      "name": {
        "localizedValue": "Managed Environment General Purpose Cores",
        "value": "ManagedEnvironmentGeneralPurposeCores"
      },
      "usage": 0
    },
    {
      "currentValue": 0,
      "limit": 200,
      "name": {
        "localizedValue": "Managed Environment Memory Optimized Cores",
        "value": "ManagedEnvironmentMemoryOptimizedCores"
      },
      "usage": 0
    },
    {
      "currentValue": 7,
      "limit": 100,
      "name": {
        "localizedValue": "Managed Environment Consumption Cores",
        "value": "ManagedEnvironmentConsumptionCores"
      },
      "usage": 7
    }
  ]
}

Also, in the portal I see this which also shows no quotas even being close:
quotas

What might be causing these errors if we are not exceeding any quotas? Or is there one that I am missing?

Interestingly, I am also noticing that even when there is only 1 message, when this error is encountered, there will be multiple jobs trying to run at the same time even though parallelism is set to 1.

Sai Prabhu Naveen Parimi 2,590 Reputation points Microsoft External Staff Moderator

2025-07-02T18:27:29.8266667+00:00

@Phillip S. Stenger Jr

Thank you for reaching out !!

Have there been any recent changes to your environment setup or resource configurations before these errors started appearing?

Are there any other applications or jobs running in the same Managed Environment that could be utilizing resources?

Can you provide more details about the configuration and deployment triggers for your job?

Phillip S. Stenger Jr 20

We haven't made any configuration changes recently.

There are other container apps running in the environment, but based on the output seen above from running:

az containerapp env list-usages --resource-group <RESOURCE_GROUP_NAME> --name <ENVIRONMENT_NAME>

none of the core limits are even close to full utilization. Moreover, this job is using a workload profile dedicated to just this job. So it shouldn't be having any issues with resource contention. Also, I should mention that this does not happen every time. Sometimes it spins up correctly.

The only thing I can think of is whether setting the min instances for the workload profile to 0 could be causing issues?

Here is the Terraform for deploying the job:

resource "azapi_resource" "main" {
  type      = "Microsoft.App/jobs@2024-10-02-preview"
  name      = var.name
  parent_id = var.resource_group_id
  identity {
    type         = "SystemAssigned, UserAssigned"
    identity_ids = [var.assigned_id]
  }
  location = var.location
  body = {
    properties = {
      configuration = {
        eventTriggerConfig = {
          parallelism            = 1
          replicaCompletionCount = 1
          scale = {
            maxExecutions   = 8
            minExecutions   = 0
            pollingInterval = 15
            rules = [{
              identity = "system"
              name     = "queue-scaling"
              type     = "azure-servicebus"
              "metadata" : {
                "queueName" : var.queue_name,
                "namespace" : var.sb_namespace_name,
                "messageCount" : "1",
                "pollingInterval" : "15",
                "cooldownPeriod" : "60"
              }
            }]
          }
          
        }
        secrets = [
          {
            name  = "appinsights-connection-string"
            value = data.azurerm_key_vault_secret.ai.value
          }
        ]
        registries = [
          {
            identity = var.assigned_id
            server   = var.acr_server
          }
        ]
        replicaRetryLimit = 3
        replicaTimeout    = 7200
        triggerType       = "Event"
      }
      environmentId = var.container_app_environment_id
      template = {
        containers = [
          {
            image = "${var.acr_server}/${var.name}:${var.container_version}"
            name  = var.name
            resources = {

              cpu    = 0.5
              memory = "1.0Gi"
            }
            env = [
              {
                name  = "DOTNET_ENVIRONMENT"
                value = var.environment
              },
              {
                name  = "BlobStorage__Uri"
                value = var.blob_storage_uri
              },
              {
                name  = "ServiceBus__Namespace"
                value = var.sb_namespace_name
              },
              {
                name = "APPLICATIONINSIGHTS_CONNECTION_STRING",
                secretRef = "appinsights-connection-string"
              },
              {
                name = "ServiceBus__QueueName"
                value = var.queue_name
              }
            ]
        }]
      }
      workloadProfileName = "[redacted]"
    }
  }
}

Sai Prabhu Naveen Parimi 2,590 Reputation points Microsoft External Staff Moderator

2025-07-02T19:23:52.5233333+00:00

@Phillip S. Stenger Jr

Thanks again for all the details.

Your suspicion about minExecutions = 0 might actually be right. When it’s set to zero, there’s no warm container running — so when a message comes in, the system has to quickly start a new one. If the workload profile is already tight on CPU (like yours with 4 vCPUs), it can lead to those “Maximum Allowed Cores exceeded” errors, even if the overall environment looks under quota.

Also, since the job is tied to a specific workload profile, it can only run on those nodes. If anything else is using a bit of CPU there, or if replicas try to spin up at once, it can hit limits fast.

Could you please give this

Set minExecutions to 1 — this keeps one replica always warm and avoids cold-start issues.

Lower maxExecutions from 8 to 4 — this reduces pressure on the profile and makes scheduling more reliable.

This should stabilize things without needing any major changes.

Here’s a helpful link that explains how quotas work in Container Apps: Container Apps quotas
Phillip S. Stenger Jr 20 Reputation points

2025-07-02T20:59:50.66+00:00

That does seem to have worked, for now at least. I change the min instances for the workflow profile to 1 and max to 2. I will keep an eye and see if the issue surfaces again.

Is this expected behavior for workload profiles that they don't reliably scale from 0? This job is not running very often so it seems like a waste to have the workload profile constantly incurring expense by running all the time.

We are not able to use the Consumption profile because when we did that in the past we were getting issues where the pods were getting pre-empted.

Phillip S. Stenger Jr 20

I tried to deploy the same solution to our prod environment and am getting the following error now:

Update environment AKS handler failed: NodePool wp10z0cb8d5 with operation type 0 deployment error is NodePool OperationType is 0 and error is Operation Failed, code : OverconstrainedZonalAllocationRequest, message : original error: Code=\"OverconstrainedZonalAllocationRequest\" Message=\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\n  - Availability Zone\\n  - Differencing (Ephemeral) Disks\\n  - Networking Constraints (such as Accelerated Networking or IPv6)\\n  - VM Size\\n\" Target=\"158\". Please note that allocation failures can also arise if a proximity placement group is associated with this VMSS.See https://learn.microsoft.com/en-us/troubleshoot/azure/azure-kubernetes/error-code-zonalallocationfailed-allocationfailed for more details.This is not AKS controlled behavior, please ask help to VMSS team for allocation failure.. Operation max retry count reached..

Sai Prabhu Naveen Parimi 2,590 Reputation points Microsoft External Staff Moderator

2025-07-03T16:14:04.3+00:00

@Phillip S. Stenger Jr

You can resolve this issue by adjusting or removing restrictive constraints like VM size, zone preference, or ephemeral disks. This is a VMSS allocation failure, not an AKS issue.

Refer to the official Microsoft troubleshooting guide for details and resolution steps: https://learn.microsoft.com/en-us/troubleshoot/azure/azure-kubernetes/error-code-zonalallocationfailed-allocationfailed
Phillip S. Stenger Jr 20 Reputation points

2025-07-03T16:18:06.1966667+00:00

Could you elaborate on how I would do that? I have to specify SKU for the workload profile. As far as I can tell, I can't set the zone preference myself nor the ephemeral disk settings.
Sai Prabhu Naveen Parimi 2,590 Reputation points Microsoft External Staff Moderator

2025-07-03T16:35:40.8233333+00:00
@Phillip S. Stenger Jr

Could you please try the below in azure CLI

az aks nodepool add \ --resource-group <rg> \ --cluster-name <aks-name> \ --name newpool \ --node-count 1 \ --node-vm-size Standard_D2s_v3 \ --zones 1 2 3 \ --os-disk-type Managed
Phillip S. Stenger Jr 20 Reputation points

2025-07-03T16:40:24.73+00:00

Will this work for a Container Apps Environment? I didn't think it was possible to interact directly with the backing Kubernetes service for Container Apps.
Krishna Chowdary Paricharla 955 Reputation points Microsoft External Staff Moderator

2025-07-09T09:45:07.1333333+00:00

Hello Phillip S. Stenger Jr •,

Could you please check the private message and provide the required details for troubleshooting the issue further.

Here is the reference link on how to access & data retention policy of private messages in Microsoft Q&A.

Your answer

Sai Prabhu Naveen Parimi 2,590 Reputation points Microsoft External Staff Moderator

2025-07-02T18:27:29.8266667+00:00

@Phillip S. Stenger Jr

Thank you for reaching out !!

Have there been any recent changes to your environment setup or resource configurations before these errors started appearing?

Are there any other applications or jobs running in the same Managed Environment that could be utilizing resources?

Can you provide more details about the configuration and deployment triggers for your job?
Sai Prabhu Naveen Parimi 2,590 Reputation points Microsoft External Staff Moderator

2025-07-02T19:23:52.5233333+00:00

@Phillip S. Stenger Jr

Thanks again for all the details.

Your suspicion about minExecutions = 0 might actually be right. When it’s set to zero, there’s no warm container running — so when a message comes in, the system has to quickly start a new one. If the workload profile is already tight on CPU (like yours with 4 vCPUs), it can lead to those “Maximum Allowed Cores exceeded” errors, even if the overall environment looks under quota.

Also, since the job is tied to a specific workload profile, it can only run on those nodes. If anything else is using a bit of CPU there, or if replicas try to spin up at once, it can hit limits fast.

Could you please give this

Set minExecutions to 1 — this keeps one replica always warm and avoids cold-start issues.

Lower maxExecutions from 8 to 4 — this reduces pressure on the profile and makes scheduling more reliable.

This should stabilize things without needing any major changes.

Here’s a helpful link that explains how quotas work in Container Apps: Container Apps quotas
Phillip S. Stenger Jr 20 Reputation points

2025-07-02T20:59:50.66+00:00

That does seem to have worked, for now at least. I change the min instances for the workflow profile to 1 and max to 2. I will keep an eye and see if the issue surfaces again.

Is this expected behavior for workload profiles that they don't reliably scale from 0? This job is not running very often so it seems like a waste to have the workload profile constantly incurring expense by running all the time.

We are not able to use the Consumption profile because when we did that in the past we were getting issues where the pods were getting pre-empted.
Phillip S. Stenger Jr 20 Reputation points

2025-07-03T14:38:31.6933333+00:00

I tried to deploy the same solution to our prod environment and am getting the following error now:

Update environment AKS handler failed: NodePool wp10z0cb8d5 with operation type 0 deployment error is NodePool OperationType is 0 and error is Operation Failed, code : OverconstrainedZonalAllocationRequest, message : original error: Code=\"OverconstrainedZonalAllocationRequest\" Message=\"Allocation failed. VM(s) with the following constraints cannot be allocated, because the condition is too restrictive. Please remove some constraints and try again. Constraints applied are:\\n - Availability Zone\\n - Differencing (Ephemeral) Disks\\n - Networking Constraints (such as Accelerated Networking or IPv6)\\n - VM Size\\n\" Target=\"158\". Please note that allocation failures can also arise if a proximity placement group is associated with this VMSS.See https://learn.microsoft.com/en-us/troubleshoot/azure/azure-kubernetes/error-code-zonalallocationfailed-allocationfailed for more details.This is not AKS controlled behavior, please ask help to VMSS team for allocation failure.. Operation max retry count reached..
Sai Prabhu Naveen Parimi 2,590 Reputation points Microsoft External Staff Moderator

2025-07-03T16:14:04.3+00:00

@Phillip S. Stenger Jr

You can resolve this issue by adjusting or removing restrictive constraints like VM size, zone preference, or ephemeral disks. This is a VMSS allocation failure, not an AKS issue.

Refer to the official Microsoft troubleshooting guide for details and resolution steps: https://learn.microsoft.com/en-us/troubleshoot/azure/azure-kubernetes/error-code-zonalallocationfailed-allocationfailed
Phillip S. Stenger Jr 20 Reputation points

2025-07-03T16:18:06.1966667+00:00

Could you elaborate on how I would do that? I have to specify SKU for the workload profile. As far as I can tell, I can't set the zone preference myself nor the ephemeral disk settings.
Sai Prabhu Naveen Parimi 2,590 Reputation points Microsoft External Staff Moderator

2025-07-03T16:35:40.8233333+00:00

@Phillip S. Stenger Jr

Could you please try the below in azure CLI

az aks nodepool add \ --resource-group <rg> \ --cluster-name <aks-name> \ --name newpool \ --node-count 1 \ --node-vm-size Standard_D2s_v3 \ --zones 1 2 3 \ --os-disk-type Managed
Phillip S. Stenger Jr 20 Reputation points

2025-07-03T16:40:24.73+00:00

Will this work for a Container Apps Environment? I didn't think it was possible to interact directly with the backing Kubernetes service for Container Apps.
Krishna Chowdary Paricharla 955 Reputation points Microsoft External Staff Moderator

2025-07-09T09:45:07.1333333+00:00

Hello Phillip S. Stenger Jr •,

Could you please check the private message and provide the required details for troubleshooting the issue further.

Here is the reference link on how to access & data retention policy of private messages in Microsoft Q&A.

Share via

Container Apps Job Maximum Allowed Cores exceeded

Your answer