Deployed Huggingface model using managed compute, cannot increase max_concurrent_requests_per_instance. Always stays 0

Question

Deployed Huggingface model using managed compute, cannot increase max_concurrent_requests_per_instance. Always stays 0

Tony Mellios 0

Hello,

I have used Azure AI foundry to deploy this model in the West US region

azureml://registries/HuggingFace/models/sentence-transformers-all-minilm-l6-v2/versions/3

I'm using and A100 instance.

Standard_NC24ads_A100_v4

The deployment is currently configured with a max concurrency of "0". I'm not really sure what this value means but we are observing that we're not getting 429's so I was trying to increase it.

{
  "app_insights_enabled": false,
  "creation_context": {
    "created_at": "2025-03-23T17:10:57.058258+00:00",
    "created_by": "Blah"
    "last_modified_at": "2025-03-23T17:10:57.058259+00:00"
  },
  "egress_public_network_access": "enabled",
  "endpoint_name": "ai-blah",
  "environment_variables": {},
  "id": "/subscriptions/BLAH/resourceGroups/rg-blah/providers/Microsoft.MachineLearningServices/workspaces/proj-blah/onlineEndpoints/ai-blah/deployments/sentence-transformers-all-min-3",
  "instance_count": 1,
  "instance_type": "Standard_NC24ads_A100_v4",
  "model": "azureml://registries/HuggingFace/models/sentence-transformers-all-minilm-l6-v2/versions/3",
  "name": "sentence-transformers-all-min-3",
  "properties": {
    "AzureAsyncOperationUri": "blah"
  },
  "provisioning_state": "Succeeded",
  "request_settings": {
    "max_concurrent_requests_per_instance": 0
  },
  "resourceGroup": "rg-blah",
  "scale_settings": {
    "type": "default"
  },
  "tags": {},
  "type": "managed"
}

I have tried amending this value to a higher one using the below command. The command returns but the value remains 0. I've tried multiple different values to no avail.

az ml online-deployment update --name sentence-transformers-all-min-3 --endpoint-name ai-hub-blah --resource-group rg-blah --workspace-name proj-blah --set request_settings.max_concurrent_requests_per_instance=4

Any help would be greatly appreciated.

Anonymous

2025-03-24T22:13:31.2833333+00:00
Hi Tony Mellios

It seems that you are experiencing an issue with setting the max_concurrent_requests_per_instance for your deployed model in Azure. The value of 0 typically indicates that the instance is set to handle requests sequentially, meaning it won't process multiple requests at the same time.

To change this value, you should ensure that you are using the correct command and that there are no restrictions or limitations on your deployment that might prevent the update. Here are some steps to follow.

Ensure that your deployment configuration allows for concurrent requests. Some configurations may have default settings that restrict concurrency.

Make sure you are using the latest version of the Azure CLI, as older versions may have bugs or lack support for certain features.

Verify that the instance type you are using supports the desired level of concurrency. Some instance types may have limitations on how many concurrent requests they can handle.

After running the update command, check for any error messages or logs that might indicate why the value is not being updated.

Can you refer these https://learn.microsoft.com/en-us/azure/machine-learning/how-to-safely-rollout-online-endpoints?view=azureml-api-2&tabs=azure-cli#scale-your-existing-deployment-to-handle-more-traffic

I hope these helps you. Thank you!
Tony Mellios 0 Reputation points

2025-03-25T15:06:19.7666667+00:00
Hi @Anonymous ,

Thanks for your note.

I do have a fresh install of the az cli that is up to date

az --version azure-cli 2.70.0 core 2.70.0 telemetry 1.1.0 Extensions: ml 2.36.0 Dependencies: msal 1.31.2b1 azure-mgmt-resource 23.1.1 Python location 'C:\Program Files\Microsoft SDKs\Azure\CLI2\python.exe' Config directory 'C:\Users\TonyMellios\.azure' Extensions directory 'C:\Users\TonyMellios\.azure\cliextensions' Python (Windows) 3.12.8 (tags/v3.12.8:2dc476b, Dec 3 2024, 19:30:04) [MSC v.1942 64 bit (AMD64)] Legal docs and information: aka.ms/AzureCliLegal Your CLI is up-to-date.

How can I verify these two points below? Can you provide instructions? I haven't been able to find anything in the documentation.

Ensure that your deployment configuration allows for concurrent requests. Some configurations may have default settings that restrict concurrency.

Verify that the instance type you are using supports the desired level of concurrency. Some instance types may have limitations on how many concurrent requests they can handle.

I am familiar with the process to instantiate more instances. In this case, I'm looking to increase concurrency within the same instance to ensure we're using the most of out the hardware. With the current concurrency, the GPU utilization remains pretty low.

Thanks!
Anonymous

2025-03-27T21:46:12.8266667+00:00
Hi Tony Mellios

Review the deployment YAML file or settings used when deploying your Hugging Face model. Look for parameters related to concurrency, such as max_concurrent_requests_per_instance. Ensure that it is set to a value greater than 0.

If using the Azure CLI, you can retrieve the current deployment configuration with the command:

az ml online-deployment show --name <deployment_name> --endpoint-name <endpoint_name>

Check the output for any settings that may restrict concurrency.

Each instance type has its own specifications, including the maximum number of concurrent requests it can handle.

You can also check the Azure portal for the instance type details or use the Azure CLI to list the available instance types and their capabilities:

az ml compute list

Look for the sku or size of your instance and verify its specifications against the Azure documentation.

By ensuring that your deployment configuration allows for concurrent requests and that your instance type supports the desired concurrency level, you should be able to optimize GPU utilization effectively.

I hope these helps you. Thank you!
Anonymous

2025-03-28T21:15:39.1066667+00:00

Hi Tony Mellios

just checking back to see if the response was helpful.
Anonymous

2025-04-01T01:02:55.0633333+00:00

Hi Tony Mellios

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution, please do share that same with the community as it can be helpful to others. Otherwise, will respond with more details and we will try to help.
Tony Mellios 0 Reputation points

2025-04-08T16:24:26.5233333+00:00

Hello,

We ended up giving up on Azure AI Foundry for models that require managed compute due to this issue. It is unclear, and undocumented, how one can increase parallelism.

Thanks for trying to help.