Submission of image segmentation jobs to existing compute cluster all of a sudden not working in Azure ML Workspace

Question

Submission of image segmentation jobs to existing compute cluster all of a sudden not working in Azure ML Workspace

Minh Tran 5

Existing python azure ml sdk code to submit jobs to a compute cluster all of a sudden stopped working. Python code prints out following error:

HttpResponseError: (UserError) The AutoMLJob input is invalid. Compute gpu-cluster not found in workspace my-workspace-name

This compute cluster - gpu-cluster does exist and I have verified that it is in a success provisioned state. The last time I was able to submit jobs to this cluster with the same code was on June 25, 2025.

When I run az ml compute list, it shows the cluster. So not sure what is causing this issue?

Minh Tran 5 Reputation points

2025-07-06T20:29:09.2266667+00:00

I have already checked my workspace numerous times. I have also tried creating a new compute cluster in code and directly using it after it was created with same issue.
Minh Tran 5 Reputation points

2025-07-06T20:32:47.9066667+00:00

Someone else also reported the same issue.

https://learn.microsoft.com/en-us/answers/questions/2338361/error-submit-the-automl-job?comment=question&translated=false#newest-question-comment

Minh Tran 5

I am working in a jupyter notebook in Azure ML Studio. Using the kernel that comes with the workspace -> Python 3.8 - Pytorch and Tensorflow.

azure-ai-ml==1.18.0


from azure.ai.ml.entities import AmlCompute
from azure.core.exceptions import ResourceNotFoundError

compute_name = "gpu-cluster"

try:
    compute = ml_client.compute.get(compute_name)
    print("Found existing compute target.")
    print(compute)
except ResourceNotFoundError:
    print("Creating a new compute target...")
    compute_config = AmlCompute(
        name=compute_name,
        type="amlcompute",
        size="Standard_NC6s_v3",
        idle_time_before_scale_down=120,
        min_instances=0,
        max_instances=4,
    )
    ml_client.begin_create_or_update(compute_config)

It outputs details of the compute cluster

Found existing compute target. enable_node_public_ip: true
id: /subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.MachineLearningServices/workspaces/<workspace>/computes/gpu-cluster idle_time_before_scale_down: 120
location: southcentralus
max_instances: 4
min_instances: 0
name: gpu-cluster
network_settings: {}
provisioning_state: Succeeded
size: Standard_NC6s_v3
ssh_public_access_enabled: true
tier: dedicated

I use the same ml_client to submit my job with that compute_name. So I know it is in the same subscription/resource_group/workspace. But error returned is the backend api can't find that compute cluster

Manas Mohanty 6,690 Reputation points Microsoft External Staff Moderator

2025-07-07T02:04:00.8033333+00:00
Hi Minh Tran

Sorry for the trouble. It seems you have marked Ameera's answer as not helpful. We appreciate your feedback

Trying to follow up for recovering sentiment.

Thank you for confirming that you are able to detect GPU cluster. The issue might be recent SDK change or backend change.

Could you do the below setting and let us know,

GPU Cluster has system system assigned identity and you have added that managed identity as storage blob reader and contributor over your storage in usage for AutoML job .

Managed identity

Recent network change -You have subnet join or read operations over virtual network for users.

Please use another GPU cluster for AutoML operations (Old Gpu might not render the change)

Debugging

You can check more details in AutoML endpoint logs (output+logs), Environment logs and take a network trace to find more leads

https://learn.microsoft.com/en-us/azure/machine-learning/how-to-troubleshoot-auto-ml?view=azureml-api-2

Please provide details requested in private chat to connect through support ticket if the issue persists.

Thank you.
Manas Mohanty 6,690 Reputation points Microsoft External Staff Moderator

2025-07-08T01:31:02.93+00:00

Hi Minh Tran

Glad to hear that you are able to use old GPUs after product group rolled backed changes.

Context - https://learn.microsoft.com/en-us/answers/questions/2338361/error-submit-the-automl-job?comment=question&translated=false#newest-question-comment

Sorry for the trouble over weekend. Requesting to be bit patient and empathetic on community members who are contributing even on weekends.

Thank you.

2 answers

Your answer

Minh Tran 5 Reputation points

2025-07-06T20:29:09.2266667+00:00

I have already checked my workspace numerous times. I have also tried creating a new compute cluster in code and directly using it after it was created with same issue.
Minh Tran 5 Reputation points

2025-07-06T20:32:47.9066667+00:00

Someone else also reported the same issue.

https://learn.microsoft.com/en-us/answers/questions/2338361/error-submit-the-automl-job?comment=question&translated=false#newest-question-comment
Minh Tran 5 Reputation points

2025-07-06T21:23:51.97+00:00

I am working in a jupyter notebook in Azure ML Studio. Using the kernel that comes with the workspace -> Python 3.8 - Pytorch and Tensorflow.

azure-ai-ml==1.18.0

from azure.ai.ml.entities import AmlCompute from azure.core.exceptions import ResourceNotFoundError compute_name = "gpu-cluster" try: compute = ml_client.compute.get(compute_name) print("Found existing compute target.") print(compute) except ResourceNotFoundError: print("Creating a new compute target...") compute_config = AmlCompute( name=compute_name, type="amlcompute", size="Standard_NC6s_v3", idle_time_before_scale_down=120, min_instances=0, max_instances=4, ) ml_client.begin_create_or_update(compute_config)

It outputs details of the compute cluster

Found existing compute target. enable_node_public_ip: true id: /subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.MachineLearningServices/workspaces/<workspace>/computes/gpu-cluster idle_time_before_scale_down: 120 location: southcentralus max_instances: 4 min_instances: 0 name: gpu-cluster network_settings: {} provisioning_state: Succeeded size: Standard_NC6s_v3 ssh_public_access_enabled: true tier: dedicated

I use the same ml_client to submit my job with that compute_name. So I know it is in the same subscription/resource_group/workspace. But error returned is the backend api can't find that compute cluster
Manas Mohanty 6,690 Reputation points Microsoft External Staff Moderator

2025-07-07T02:04:00.8033333+00:00

Hi Minh Tran

Sorry for the trouble. It seems you have marked Ameera's answer as not helpful. We appreciate your feedback

Trying to follow up for recovering sentiment.

Thank you for confirming that you are able to detect GPU cluster. The issue might be recent SDK change or backend change.

Could you do the below setting and let us know,

GPU Cluster has system system assigned identity and you have added that managed identity as storage blob reader and contributor over your storage in usage for AutoML job .

Managed identity

Recent network change -You have subnet join or read operations over virtual network for users.

Please use another GPU cluster for AutoML operations (Old Gpu might not render the change)

Debugging

You can check more details in AutoML endpoint logs (output+logs), Environment logs and take a network trace to find more leads

https://learn.microsoft.com/en-us/azure/machine-learning/how-to-troubleshoot-auto-ml?view=azureml-api-2

Please provide details requested in private chat to connect through support ticket if the issue persists.

Thank you.
Manas Mohanty 6,690 Reputation points Microsoft External Staff Moderator

2025-07-08T01:31:02.93+00:00

Hi Minh Tran

Glad to hear that you are able to use old GPUs after product group rolled backed changes.

Context - https://learn.microsoft.com/en-us/answers/questions/2338361/error-submit-the-automl-job?comment=question&translated=false#newest-question-comment

Sorry for the trouble over weekend. Requesting to be bit patient and empathetic on community members who are contributing even on weekends.

Thank you.

Answer 1

Hello !

Thank you for posting on Microsoft Learn.

Have you checked if that the workspace object in your Python code points to the correct subscription, resource group, and workspace name ?

Try to print the workspace details in your code:

print(ws.name, ws.location, ws.resource_group)

You might be authenticating in a different Azure subscription or workspace than where gpu-cluster exists.

There could have been a breaking change in the azureml-sdk or azure-ai-ml version if you recently upgraded or if a managed compute environment updated.

Try to check your current version:

pip show azure-ai-ml
pip show azureml-sdk

If using azure-ai-ml, verify if you're using a compatible version (>=1.15.0) and match the API behavior with:

from azure.ai.ml import MLClient

If the above doesn’t help, try re-creating the compute cluster with a different name and update your code to reference the new cluster. This can help isolate the issue to a potential internal registration bug.

If your issue is still persisting, raise a support ticket with Azure (include your subscription ID, workspace name, cluster name, and timestamp of error).

Answer 2

Minh Tran 5

This was caused by an errant deployment by Azure team. It was resolved by user who reported the same issue previously mentioned by submitting a support ticket that resulted in rollback of errant code.

https://learn.microsoft.com/en-us/answers/questions/2338361/error-submit-the-automl-job?comment=question&translated=false#newest-question-comment

Share via

Submission of image segmentation jobs to existing compute cluster all of a sudden not working in Azure ML Workspace

2 answers

Your answer