Deployment of Multiple Models to Container Instance Fails in Azure DevOps

Question

Hi Team,

I am trying to deploy 2 ML models ( which is registered in Model Registry ) to Azure Container Instance using DevOps Release pipeline using AZ CLI ML extension

My ACI Configuration is :

containerResourceRequirements: cpu: 1 memoryInGB: 4 computeType: ACI

Inference Config :

entryScript: score.py runtime: python condaFile: conda_dependencies.yml extraDockerfileSteps: schemaFile: sourceDirectory: enableGpu: False baseImage: baseImageRegistry:

All score.py, conda_dependencies.yml, aciDeploymentConfig.yml is placed in a flattened directory which is publised in to DevOps pipeline artifcat and looks like

DevOps Deploy command looks like

az ml model deploy -g $(ml.resourceGroup) -w $(ml.workspace) --name $(service.name.staging) -f ./model.json -m "GloVe:4" --dc aciDeploymentConfig.yml --ic inferenceConfig.yml --overwrite --debug

Also i have set the working directory as the folder where all above files are placed. something like

$(System.DefaultWorkingDirectory)/_Symptom-Code-Indexing/symptom_model/a

Its getting in to an exception as

2020-06-08T12:50:27.9202657Z "error": {
2020-06-08T12:50:27.9208361Z "message": "Received bad response from Model Management Service:\nResponse Code: 400\nHeaders: {'Date': 'Mon, 08 Jun 2020 12:50:27 GMT', 'Content-Type': 'application/json', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'Request-Context': 'appId=cid-v1:2d2e8e63-272e-4b3c-8598-4ee570a0e70d', 'x-ms-client-request-id': '823e8483923846b1958c08ffaba074ff', 'x-ms-client-session-id': '62e84a29-b77c-456f-9d91-5ca6be26f79c', 'api-supported-versions': '1.0, 2018-03-01-preview, 2018-11-19', 'Strict-Transport-Security': 'max-age=15724800; includeSubDomains; preload'}\nContent: b'{"code":"BadRequest","statusCode":400,"message":"The request is invalid.","details":[{"code":"InvalidOverwriteRequest","message":"Invalid overwrite request - cannot update container resource requirements, dns name label, or deployment type. Please delete and redeploy this service."}],"correlation":{"RequestId":"823e8483923846b1958c08ffaba074ff"}}'"
2020-06-08T12:50:27.9212109Z }
2020-06-08T12:50:27.9212376Z }}
2020-06-08T12:50:27.9213437Z {'Azure-cli-ml Version': '1.6.0', 'Error': WebserviceException:
2020-06-08T12:50:27.9214158Z Message: Received bad response from Model Management Service:
2020-06-08T12:50:27.9214688Z Response Code: 400
2020-06-08T12:50:27.9217800Z Headers: {'Date': 'Mon, 08 Jun 2020 12:50:27 GMT', 'Content-Type': 'application/json', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'Request-Context': 'appId=cid-v1:2d2e8e63-272e-4b3c-8598-4ee570a0e70d', 'x-ms-client-request-id': '823e8483923846b1958c08ffaba074ff', 'x-ms-client-session-id': '62e84a29-b77c-456f-9d91-5ca6be26f79c', 'api-supported-versions': '1.0, 2018-03-01-preview, 2018-11-19', 'Strict-Transport-Security': 'max-age=15724800; includeSubDomains; preload'}
2020-06-08T12:50:27.9222115Z Content: b'{"code":"BadRequest","statusCode":400,"message":"The request is invalid.","details":[{"code":"InvalidOverwriteRequest","message":"Invalid overwrite request - cannot update container resource requirements, dns name label, or deployment type. Please delete and redeploy this service."}],"correlation":{"RequestId":"823e8483923846b1958c08ffaba074ff"}}'
2020-06-08T12:50:27.9223705Z InnerException None
2020-06-08T12:50:27.9224049Z ErrorResponse
2020-06-08T12:50:27.9224320Z {
2020-06-08T12:50:27.9224617Z "error": {
2020-06-08T12:50:27.9229025Z "message": "Received bad response from Model Management Service:\nResponse Code: 400\nHeaders: {'Date': 'Mon, 08 Jun 2020 12:50:27 GMT', 'Content-Type': 'application/json', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'Request-Context': 'appId=cid-v1:2d2e8e63-272e-4b3c-8598-4ee570a0e70d', 'x-ms-client-request-id': '823e8483923846b1958c08ffaba074ff', 'x-ms-client-session-id': '62e84a29-b77c-456f-9d91-5ca6be26f79c', 'api-supported-versions': '1.0, 2018-03-01-preview, 2018-11-19', 'Strict-Transport-Security': 'max-age=15724800; includeSubDomains; preload'}\nContent: b'{"code":"BadRequest","statusCode":400,"message":"The request is invalid.","details":[{"code":"InvalidOverwriteRequest","message":"Invalid overwrite request - cannot update container resource requirements, dns name label, or deployment type. Please delete and redeploy this service."}],"correlation":{"RequestId":"823e8483923846b1958c08ffaba074ff"}}'"
2020-06-08T12:50:27.9230782Z }
2020-06-08T12:50:27.9230908Z }}
2020-06-08T12:50:27.9231134Z Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x7fea2ca1f730>]
2020-06-08T12:50:27.9231431Z az_command_data_logger : exit code: 1
2020-06-08T12:50:27.9275693Z telemetry.save : Save telemetry record of length 7390 in cache
2020-06-08T12:50:27.9280735Z telemetry.check : Negative: The /home/vsts/work/_temp/.azclitask/telemetry.txt was modified at 2020-06-08 12:47:41.161160, which in less than 600.000000 s
2020-06-08T12:50:27.9290480Z command ran in 55.735 seconds.
2020-06-08T12:50:28.1525434Z ##[error]Script failed with exit code: 1
2020-06-08T12:50:28.1536650Z [command]/opt/hostedtoolcache/Python/3.6.10/x64/bin/az account clear
2020-06-08T12:50:29.9078943Z ##[section]Finishing: Deploy Model to ACI

But when i tried to Deploy it using Python SDK it works as well. Is there any permission issues or login to be set before using DevOps Release. I have not done any sort of login in my DevOps Build pipeline.

Any pointers on what is going wrong here ? It would be really helpful.

Thanks,
Srijith

Answer 1

They're actively answering Devops question in dedicated forums here.

https://developercommunity.visualstudio.com/spaces/21/index.html

--please don't forget to Accept as answer if the reply is helpful--

---

Regards, Dave Patrick ....
Microsoft Certified Professional
Microsoft MVP [Windows Server] Datacenter Management

Disclaimer: This posting is provided "AS IS" with no warranties or guarantees, and confers no rights.