Submission of image segmentation jobs to existing compute cluster all of a sudden not working in Azure ML Workspace

I have already checked my workspace numerous times. I have also tried creating a new compute cluster in code and directly using it after it was created with same issue.

Someone else also reported the same issue.

https://learn.microsoft.com/en-us/answers/questions/2338361/error-submit-the-automl-job?comment=question&translated=false#newest-question-comment

I am working in a jupyter notebook in Azure ML Studio. Using the kernel that comes with the workspace -> Python 3.8 - Pytorch and Tensorflow.

azure-ai-ml==1.18.0

from azure.ai.ml.entities import AmlCompute
from azure.core.exceptions import ResourceNotFoundError

compute_name = "gpu-cluster"

try:
    compute = ml_client.compute.get(compute_name)
    print("Found existing compute target.")
    print(compute)
except ResourceNotFoundError:
    print("Creating a new compute target...")
    compute_config = AmlCompute(
        name=compute_name,
        type="amlcompute",
        size="Standard_NC6s_v3",
        idle_time_before_scale_down=120,
        min_instances=0,
        max_instances=4,
    )
    ml_client.begin_create_or_update(compute_config)

It outputs details of the compute cluster

Found existing compute target. enable_node_public_ip: true
id: /subscriptions/<subscription_id>/resourceGroups/<resource_group>/providers/Microsoft.MachineLearningServices/workspaces/<workspace>/computes/gpu-cluster idle_time_before_scale_down: 120
location: southcentralus
max_instances: 4
min_instances: 0
name: gpu-cluster
network_settings: {}
provisioning_state: Succeeded
size: Standard_NC6s_v3
ssh_public_access_enabled: true
tier: dedicated

I use the same ml_client to submit my job with that compute_name. So I know it is in the same subscription/resource_group/workspace. But error returned is the backend api can't find that compute cluster

Hi Minh Tran

Sorry for the trouble. It seems you have marked Ameera's answer as not helpful. We appreciate your feedback

Trying to follow up for recovering sentiment.

Thank you for confirming that you are able to detect GPU cluster. The issue might be recent SDK change or backend change.

Could you do the below setting and let us know,

1. GPU Cluster has system system assigned identity and you have added that managed identity as storage blob reader and contributor over your storage in usage for AutoML job .

Managed identity

1. Recent network change -You have subnet join or read operations over virtual network for users.
2. Please use another GPU cluster for AutoML operations (Old Gpu might not render the change)

Debugging

You can check more details in AutoML endpoint logs (output+logs), Environment logs and take a network trace to find more leads

https://learn.microsoft.com/en-us/azure/machine-learning/how-to-troubleshoot-auto-ml?view=azureml-api-2

Please provide details requested in private chat to connect through support ticket if the issue persists.

Thank you.