Issues with Azure Foundry deployment

Question

Hi all,

I am currently facing an issue when I deploy a qwen model on managed compute. It successfully deployed but I am receiving the problem "ERR_BAD_RESPONSEThe service hit an internal error to process the request. Sorry about that." when I send a prompt in the playground.

Thank you

Answer 1

Hello Randy Sim,

Greetings! Thanks for raising this question in the Q&A forum.

ERR_BAD_RESPONSE in the Foundry playground is a generic front-end error meaning the playground got back something other than a valid completion from your deployment's backend, it does not by itself tell you whether the cause is authentication, capacity, a crashed container, or the model still warming up. Since the deployment shows as successfully provisioned but fails on the first real inference call, the most common root causes for Qwen on managed compute are the deployment still finishing model load after the provisioning state turned "Succeeded," an accelerator or deployment template mismatch causing an out-of-memory failure inside the container, or the container crashing on first request and needing a manual restart. Provisioning success only means the infrastructure came up, it does not guarantee the model server inside finished loading weights.

Confirm the deployment is actually healthy, not just provisioned

In Foundry, go to Models + endpoints, select your Qwen deployment, and check its live status rather than relying on the initial "deployed" confirmation. For managed compute deployments backed by Azure Machine Learning online endpoints, provisioning state and traffic health are two separate signals. If you deployed via SDK or CLI, confirm directly:

az ml online-deployment show --name <deploymentName> --endpoint-name <endpointName> --resource-group <rgName> --workspace-name <projectName> --query "{provisioningState:provisioningState, appInsightsEnabled:appInsightsEnabled}"

Pull the deployment logs to see the actual backend exception

The playground error message deliberately hides the real error. Go to the deployment's Logs tab in Foundry, or query directly:

az ml online-deployment get-logs --name <deploymentName> --endpoint-name <endpointName> --resource-group <rgName> --workspace-name <projectName> --lines 200

Look specifically for CUDA out of memory, OOMKilled, container restart messages, or a stack trace from the scoring server. An out-of-memory error here confirms the accelerator template you picked doesn't have enough GPU memory for the specific Qwen variant and context length you selected.

Match the deployment template to the model size you actually deployed

Managed compute in Foundry requires you to explicitly pick a deployment template and accelerator type (A100, H100, or MI300X) alongside the model, and templates are tied to specific GPU counts and context lengths. If you picked a smaller Qwen variant's template for a larger one, or a single-accelerator template with a large context length setting, the container can crash under real inference load even though the deployment itself came up. In the portal, reselect Deploy on the model card and confirm the template name matches the exact Qwen model size and variant you intended, for example a qwen--qwen3-32b template should not be paired with a qwen3-72b model.

Bypass the playground and call the endpoint directly

This isolates whether the problem is playground-specific or backend-wide, and gives you the real HTTP status code and error body instead of the generic playground message.

curl -X POST "https://<your-foundry-endpoint>.services.ai.azure.com/models/chat/completions?api-version=2024-05-01-preview" \
  -H "Content-Type: application/json" \
  -H "api-key: <your-api-key>" \
  -d '{"model": "<your-deployment-name>", "messages": [{"role": "user", "content": "hello"}]}'

If this also fails, note the exact status code and the x-ms-request-id or apim-request-id response header, you will need this for support.

Check accelerator quota for the family you selected

Quota for managed compute is tracked per accelerator family, such as H100_80GB or A100_80GB, separately from your general Azure VM quota. If your subscription only had partial quota approved, the endpoint can come up with fewer replicas than the template expects, causing intermittent internal errors under any load. Check this from Management center > Quota in Foundry for the accelerator family tied to your deployment.

If logs show no clear error, escalate with the request ID

If the container logs show no exception and the deployment reports healthy, this points to a backend routing or capacity issue that only the Foundry service team can see from their side. Open a support request:

• Issue type: Technical
  • Service: Microsoft Foundry
    • Problem type: Model deployment and inference
    Include your Foundry resource name, the deployment name, the exact Qwen model and deployment template used, the timestamp of a failed request, and the apim-request-id from step 4 if you were able to capture one.

If this answer helps you kindly accept the answer which will help others who have similar questions.

Best Regards,

Jerald Felix.