How to access deployed container and edit score script

He Cheng Hui 0

I used Azure ML studio to set up a custom environment (base image using mcr.microsoft.com/azureml/openmpi4.1.0-cuda11.6-cudnn8-ubuntu20.04) and an online endpoint using that env.

The endpoint managed to deploy successfully using a Standard_NC4as_T4_v3 SKU. I would like to:

Access the container to check if torch is accessing the GPU using terminal
Edit the score script

I am able to do so locally but how do I do so after deploying online?

Ramr-msft 17,616 Reputation points

2023-03-17T01:50:15.5+00:00

@He Cheng Hui Here is the Doc to update the deployed online endpoint using the SDK.

And if you are simply trying to debug and redeploy, doing so locally Please follow the below doc. https://learn.microsoft.com/en-us/azure/machine-learning/how-to-troubleshoot-deployment#debug-locally

He Cheng Hui 0

@Ramr-msft

Thank you. I have tried following the Doc to update the online deployment but I got the following error:

#Creating a local endpoint
from azure.ai.ml.entities import (
    ManagedOnlineEndpoint,
    ManagedOnlineDeployment,
    Model,
    Environment,
    CodeConfiguration,
)

ml_client = MLClient(interactive_auth, subscription_id, resource_group, workspace)

deployment = ManagedOnlineDeployment(
    name=deployment_name,
    endpoint_name=endpoint_name,
    code_configuration=CodeConfiguration(
        code=path, scoring_script="score_test.py"
    )
)

endpoint = ManagedOnlineEndpoint(name=endpoint_name)
deployment = ml_client.online_deployments.begin_create_or_update(deployment)

>>> Check: endpoint <name> exists
>>> Uploading ML (0.03 MBs): 100%|################################################################################################################################################################################################| 28627/28627 [00:00<00:00, 31116.31it/s]
>>> 
File "\Anaconda\envs\Azure_test\lib\site-packages\azure\ai\ml\_restclient\v2022_02_01_preview\operations\_online_deployments_operations.py", line 827, in _create_or_update_initial
    raise HttpResponseError(response=response, error_format=ARMErrorFormat)
azure.core.exceptions.HttpResponseError: (BadRequest) The request is invalid.
Code: BadRequest
Message: The request is invalid.
Exception Details:      (EmptyOrInvalidParameter) Number of replicas should be set or auto scaler should be turned on
        Code: EmptyOrInvalidParameter
        Message: Number of replicas should be set or auto scaler should be turned on
Additional Information:Type: ComponentName
Info: {
    "value": "managementfrontend"
}Type: Correlation
Info: {
    "value": {
        "operation": "c4993bf5e3df7f1a402ce24329f2f279",
        "request": "f89a34d2dc0fcaa3"
    }
}Type: Environment
Info: {
    "value": "southeastasia"
}Type: Location
Info: {
    "value": "southeastasia"
}Type: Time
Info: {
    "value": "2023-03-17T07:25:55.4021795+00:00"
}

I have tried doing locally first but i wasn't able to get the container to use my GPU (window host, linux container using this).

Ramr-msft 17,616 Reputation points

2023-04-04T11:09:12.11+00:00

@He Cheng Hui Thanks for the details. The error message suggests that the number of replicas should be set or the auto scaler should be turned on. Here is the troubleshooting document for deployment.