Azure ml cli - Deploy and score a machine learning model by using an online endpoint issue

Question

Azure ml cli - Deploy and score a machine learning model by using an online endpoint issue

Carla Marques Fiadeiro 46

Hi,

I'm using Azure DevOps pipelines with azure cli tasks and I'm trying to deploy and score a machine learning model by using an online endpoint.
I'm following the steps here: https://learn.microsoft.com/en-us/azure/machine-learning/how-to-deploy-online-endpoints?tabs=azure-cli

So I have my endpoint.yml file like this:

$schema: https://azuremlschemas.azureedge.net/latest/managedOnlineEndpoint.schema.json
name: my-test-endpoint
auth_mode: key

And using azure cli with the following syntax:

az ml online-endpoint create --resource-group $(azureml.resourceGroup) --workspace-name $(azureml.workspaceName) -n $(endpoint.name) -f $(endpoint.file)

where endpoint.file is endpoint.yml and endpoint.name is my-test-endpoint.

But everytime that I run it I get the following error:

2022-12-23T14:16:35.6220095Z Traceback (most recent call last):

2022-12-23T14:16:35.6220861Z File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/entities/_util.py", line 147, in load_from_dict
2022-12-23T14:16:35.6221661Z return schema(context=context).load(data, **kwargs)
2022-12-23T14:16:35.6222337Z File "/opt/az/azcliextensions/ml/marshmallow/schema.py", line 722, in load
2022-12-23T14:16:35.6223207Z return self._do_load(
2022-12-23T14:16:35.6223828Z File "/opt/az/azcliextensions/ml/marshmallow/schema.py", line 909, in _do_load
2022-12-23T14:16:35.6224434Z raise exc
2022-12-23T14:16:35.6226674Z marshmallow.exceptions.ValidationError: {'baseImage': ['Unknown field.'], 'sourceDirectory': ['Unknown field.'], 'runtime': ['Unknown field.'], 'baseImageRegistry': ['Unknown field.'], 'enableGpu': ['Unknown field.'], 'entryScript': ['Unknown field.'], 'condaFile': ['Unknown field.'], 'extraDockerfileSteps': ['Unknown field.'], 'schemaFile': ['Unknown field.']}
2022-12-23T14:16:35.6227645Z
2022-12-23T14:16:35.6228137Z During handling of the above exception, another exception occurred:
2022-12-23T14:16:35.6228506Z
2022-12-23T14:16:35.6228910Z Traceback (most recent call last):
2022-12-23T14:16:35.6229617Z File "/opt/az/lib/python3.10/site-packages/knack/cli.py", line 233, in invoke
2022-12-23T14:16:35.6230227Z cmd_result = self.invocation.execute(args)
2022-12-23T14:16:35.6230998Z File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/init.py", line 663, in execute
2022-12-23T14:16:35.6231571Z raise ex
2022-12-23T14:16:35.6232294Z File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/init.py", line 726, in _run_jobs_serially
2022-12-23T14:16:35.6232992Z results.append(self._run_job(expanded_arg, cmd_copy))
2022-12-23T14:16:35.6233790Z File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/init.py", line 697, in _run_job
2022-12-23T14:16:35.6234384Z result = cmd_copy(params)
2022-12-23T14:16:35.6235123Z File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/init.py", line 333, in call
2022-12-23T14:16:35.6235753Z return self.handler(*args, **kwargs)
2022-12-23T14:16:35.6236523Z File "/opt/az/lib/python3.10/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
2022-12-23T14:16:35.6237306Z return op(**command_args)
2022-12-23T14:16:35.6237940Z File "/opt/az/azcliextensions/ml/azext_mlv2/manual/custom/online_endpoint.py", line 119, in ml_online_endpoint_create
2022-12-23T14:16:35.6238660Z log_and_raise_error(err, debug, yaml_operation=yaml_operation)
2022-12-23T14:16:35.6239353Z File "/opt/az/azcliextensions/ml/azext_mlv2/manual/custom/raise_error.py", line 145, in log_and_raise_error
2022-12-23T14:16:35.6239931Z raise cli_error
2022-12-23T14:16:35.6240549Z File "/opt/az/azcliextensions/ml/azext_mlv2/manual/custom/online_endpoint.py", line 87, in ml_online_endpoint_create
2022-12-23T14:16:35.6241287Z endpoint = load_online_endpoint(source=file, params_override=params_override)
2022-12-23T14:16:35.6242072Z File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/entities/_load_functions.py", line 571, in load_online_endpoint
2022-12-23T14:16:35.6242862Z return load_common(OnlineEndpoint, source, relative_origin, **kwargs)
2022-12-23T14:16:35.6243636Z File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/entities/_load_functions.py", line 105, in load_common
2022-12-23T14:16:35.6244326Z raise e
2022-12-23T14:16:35.6244966Z File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/entities/_load_functions.py", line 89, in load_common
2022-12-23T14:16:35.6245799Z return _load_common_raising_marshmallow_error(cls, yaml_dict, relative_origin, params_override, **kwargs)
2022-12-23T14:16:35.6246693Z File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/entities/_load_functions.py", line 135, in _load_common_raising_marshmallow_error
2022-12-23T14:16:35.6247733Z return cls._load(data=yaml_dict, yaml_path=relative_origin, params_override=params_override, **kwargs)
2022-12-23T14:16:35.6248562Z File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/entities/_endpoint/online_endpoint.py", line 264, in _load
2022-12-23T14:16:35.6249587Z return load_from_dict(ManagedOnlineEndpointSchema, data, context)
2022-12-23T14:16:35.6250476Z File "/opt/az/azcliextensions/ml/azext_mlv2/manual/vendored_curated_sdk/azure/ai/ml/entities/_util.py", line 150, in load_from_dict
2022-12-23T14:16:35.6251261Z raise ValidationError(decorate_validation_error(schema, pretty_error, additional_message))
2022-12-23T14:16:35.6251994Z marshmallow.exceptions.ValidationError: Validation for ManagedOnlineEndpointSchema failed:
2022-12-23T14:16:35.6252408Z
2022-12-23T14:16:35.6252773Z {
2022-12-23T14:16:35.6253144Z "baseImage": [
2022-12-23T14:16:35.6253577Z "Unknown field."
2022-12-23T14:16:35.6253956Z ],
2022-12-23T14:16:35.6254339Z "sourceDirectory": [
2022-12-23T14:16:35.6254766Z "Unknown field."
2022-12-23T14:16:35.6255160Z ],
2022-12-23T14:16:35.6255526Z "runtime": [
2022-12-23T14:16:35.6255920Z "Unknown field."
2022-12-23T14:16:35.6256329Z ],
2022-12-23T14:16:35.6256715Z "baseImageRegistry": [
2022-12-23T14:16:35.6257139Z "Unknown field."
2022-12-23T14:16:35.6257551Z ],
2022-12-23T14:16:35.6257922Z "enableGpu": [
2022-12-23T14:16:35.6258319Z "Unknown field."
2022-12-23T14:16:35.6258693Z ],
2022-12-23T14:16:35.6259099Z "entryScript": [
2022-12-23T14:16:35.6259737Z "Unknown field."
2022-12-23T14:16:35.6260924Z ],
2022-12-23T14:16:35.6261162Z "condaFile": [
2022-12-23T14:16:35.6261444Z "Unknown field."
2022-12-23T14:16:35.6261684Z ],
2022-12-23T14:16:35.6261953Z "extraDockerfileSteps": [
2022-12-23T14:16:35.6262243Z "Unknown field."
2022-12-23T14:16:35.6262502Z ],
2022-12-23T14:16:35.6262734Z "schemaFile": [
2022-12-23T14:16:35.6263014Z "Unknown field."
2022-12-23T14:16:35.6263251Z ]
2022-12-23T14:16:35.6263472Z }
2022-12-23T14:16:35.6263564Z
2022-12-23T14:16:35.6263782Z
2022-12-23T14:16:35.6264425Z Visit this link to refer to the OnlineEndpoint schema if needed: https://aka.ms/ml-cli-v2-endpoint-online-yaml-reference.

Blockquote

It seems some validations are failing and it's complaining about some fields which I have no idea what are about (my YAML file doesn't have those anywhere).

I've also checked https://azuremlschemas.azureedge.net/latest/managedOnlineEndpoint.schema.json and I can't see any reference to those there.

Can you help me understanding what's going on?

Thank you in advance

Rohit Mungi 49,131 Reputation points Microsoft Employee Moderator

2022-12-26T06:50:38.583+00:00

@Anonymous Is the variable $(endpoint.file) passing the relative path or the complete path of the YAML file containing the Azure ML online-endpoint specification?
I think it might be passing the relative path and when the command runs through your DevOps pipeline it is unable to load the file. Try using the complete path or run the same command locally and check if it works?
Carla Marques Fiadeiro 46 Reputation points

2022-12-26T10:28:02.357+00:00

Hi @Rohit Mungi ,

First of all, thank you for the reply.

I've tried with both relative and complete path and I'm still getting the same error.
Also running it locally, I get the same issue :(

Thank you,
Carla
Rohit Mungi 49,131 Reputation points Microsoft Employee Moderator

2022-12-26T11:20:37.607+00:00

Which version of the ml extension are you using? I think it could be an issue with the setup or the version of the extension.
az extension show --name ml should display the version. I am using version 2.6.1 and with the same command and YAML file the endpoint was created successfully.
Carla Marques Fiadeiro 46 Reputation points

2022-12-26T16:03:06.657+00:00

Hi @Rohit Mungi ,

I'm using the lastest version of it, I think.

This is what I get when it comes to version:

What type of compute did you tried? Managed?

Thank you,
Carla
Rohit Mungi 49,131 Reputation points Microsoft Employee Moderator

2022-12-27T12:10:06.383+00:00

I have upgraded to your version and the managed endpoint creates successfully. Is your workspace secured i.e does it use private endpoint or virtual network?