Azure Machine Learning
An Azure machine learning service for building and deploying models.
The problem was with the image build process, the dependencies weren't installed properly.
How to properly use Azure Machine Learning environments when I have to use a curated environment based on a custom image?
![](data:image/svg+xml, %3Csvg xmlns='http://www.w3.org/2000/svg' height='64' class='font-weight-bold' style='font: 600 30.11764705882353px "SegoeUI", Arial' width='64'%3E%3Ccircle fill='hsl(224.00000000000003, 80%, 31%)' cx='32' cy='32' r='32' /%3E%3Ctext x='50%25' y='55%25' dominant-baseline='middle' text-anchor='middle' fill='%23FFF' %3EBP%3C/text%3E%3C/svg%3E)
Balla Péter
5
Reputation points
I am building and a docker image in an Azure Devops pipeline which is based on the mcr.microsoft.com/azureml/openmpi4.1.0-ubuntu20.04 base image recommended by Azure ML. I push the docker image to Azure ML ACR, and try to access it from there during environment setup.
I use the following environment description file:
$schema: https://azuremlschemas.azureedge.net/latest/environment.schema.json
name: name-of-environment
image: nameofmyacr.azurecr.io/my_repo/my-image:my_tag
os_type: linux
And initiate the environment creation with this az cli command:
az ml environment create \
--resource-group rg_name \
--workspace-name ws_name \
--version env_version \
--file /path/to/file/environment.yaml
During the source image build, I install a bunch of required python packages with pip (without conda), including this one:
azureml-inference-server-http
The environment setup itself succeeds, yet when I try to use it during a deployment creation, I get the following error:
Instance status:
SystemSetup: Succeeded
UserContainerImagePull: Succeeded
ModelDownload: Succeeded
UserContainerStart: InProgress
Container events:
Kind: Pod, Name: Downloading, Type: Normal, Time: 2024-10-10T16:10:40.285553Z, Message: Start downloading models
Kind: Pod, Name: Pulling, Type: Normal, Time: 2024-10-10T16:10:40.67945Z, Message: Start pulling container image
Kind: Pod, Name: Pulled, Type: Normal, Time: 2024-10-10T16:11:23.453312Z, Message: Container image is pulled successfully
Kind: Pod, Name: Downloaded, Type: Normal, Time: 2024-10-10T16:11:23.453312Z, Message: Models are downloaded successfully
Kind: Pod, Name: Created, Type: Normal, Time: 2024-10-10T16:11:23.67159Z, Message: Created container inference-server
Kind: Pod, Name: Started, Type: Normal, Time: 2024-10-10T16:11:23.74798Z, Message: Started container inference-server
Container logs:
2024-10-10T16:11:23,726359085+00:00 - rsyslog/run
2024-10-10T16:11:23,727408496+00:00 - nginx/run
2024-10-10T16:11:23,731947543+00:00 - gunicorn/run
2024-10-10T16:11:23,734036564+00:00 | gunicorn/run |
2024-10-10T16:11:23,736560690+00:00 | gunicorn/run | ###############################################
2024-10-10T16:11:23,738069606+00:00 | gunicorn/run | AzureML Container Runtime Information
2024-10-10T16:11:23,739378719+00:00 | gunicorn/run | ###############################################
2024-10-10T16:11:23,740919835+00:00 | gunicorn/run |
2024-10-10T16:11:24,903816414+00:00 | gunicorn/run |
2024-10-10T16:11:24,907330750+00:00 | gunicorn/run | AzureML image information: openmpi4.1.0-ubuntu20.04, Materializaton Build:20240918.v1
2024-10-10T16:11:24,908902567+00:00 | gunicorn/run |
2024-10-10T16:11:24,910450682+00:00 | gunicorn/run |
2024-10-10T16:11:24,912047799+00:00 | gunicorn/run | PATH environment variable: /opt/miniconda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2024-10-10T16:11:24,913501114+00:00 | gunicorn/run | PYTHONPATH environment variable:
2024-10-10T16:11:24,914855128+00:00 | gunicorn/run |
2024-10-10T16:11:25,224749820+00:00 | gunicorn/run | CONDAPATH environment variable: /opt/miniconda
# conda environments:
#
base /opt/miniconda
2024-10-10T16:11:25,769083827+00:00 | gunicorn/run |
2024-10-10T16:11:25,770500142+00:00 | gunicorn/run | Pip Dependencies (before dynamic installation)
archspec @ file:///home/conda/feedstock_root/build_artifacts/archspec_1708969572489/work
boltons @ file:///home/conda/feedstock_root/build_artifacts/boltons_1711936407380/work
Brotli @ file:///home/conda/feedstock_root/build_artifacts/brotli-split_1725267488082/work
certifi @ file:///home/conda/feedstock_root/build_artifacts/certifi_1725278078093/work/certifi
cffi @ file:///home/conda/feedstock_root/build_artifacts/cffi_1725560520483/work
charset-normalizer @ file:///home/conda/feedstock_root/build_artifacts/charset-normalizer_1698833585322/work
colorama @ file:///home/conda/feedstock_root/build_artifacts/colorama_1666700638685/work
conda @ file:///home/conda/feedstock_root/build_artifacts/conda_1722273159603/work
conda-content-trust @ file:///home/conda/feedstock_root/build_artifacts/conda-content-trust_1693490762241/work
conda-libmamba-solver @ file:///home/conda/feedstock_root/build_artifacts/conda-libmamba-solver_1721292473987/work/src
conda-package-handling @ file:///home/conda/feedstock_root/build_artifacts/conda-package-handling_1717678605937/work
conda_package_streaming @ file:///home/conda/feedstock_root/build_artifacts/conda-package-streaming_1717678526951/work
cryptography @ file:///home/conda/feedstock_root/build_artifacts/cryptography-split_1725443066932/work
distro @ file:///home/conda/feedstock_root/build_artifacts/distro_1704321475663/work
frozendict @ file:///home/conda/feedstock_root/build_artifacts/frozendict_1715092766944/work
h2 @ file:///home/conda/feedstock_root/build_artifacts/h2_1634280454336/work
hpack==4.0.0
hyperframe @ file:///home/conda/feedstock_root/build_artifacts/hyperframe_1619110129307/work
idna @ file:///home/conda/feedstock_root/build_artifacts/idna_1726459485162/work
jsonpatch @ file:///home/conda/feedstock_root/build_artifacts/jsonpatch_1695536281965/work
jsonpointer @ file:///home/conda/feedstock_root/build_artifacts/jsonpointer_1725302897999/work
libmambapy @ file:///home/conda/feedstock_root/build_artifacts/mamba-split_1725066159413/work/libmambapy
menuinst @ file:///home/conda/feedstock_root/build_artifacts/menuinst_1725358985579/work
packaging @ file:///home/conda/feedstock_root/build_artifacts/packaging_1718189413536/work
platformdirs @ file:///home/conda/feedstock_root/build_artifacts/platformdirs_1726613481435/work
pluggy @ file:///home/conda/feedstock_root/build_artifacts/pluggy_1713667077545/work
pycosat @ file:///home/conda/feedstock_root/build_artifacts/pycosat_1696355758174/work
pycparser @ file:///home/conda/feedstock_root/build_artifacts/pycparser_1711811537435/work
pyOpenSSL @ file:///home/conda/feedstock_root/build_artifacts/pyopenssl_1722587090966/work
PySocks @ file:///home/conda/feedstock_root/build_artifacts/pysocks_1661604839144/work
requests @ file:///home/conda/feedstock_root/build_artifacts/requests_1717057054362/work
ruamel.yaml @ file:///home/conda/feedstock_root/build_artifacts/ruamel.yaml_1707298115475/work
ruamel.yaml.clib @ file:///home/conda/feedstock_root/build_artifacts/ruamel.yaml.clib_1707314473442/work
tqdm @ file:///home/conda/feedstock_root/build_artifacts/tqdm_1722737464726/work
truststore @ file:///home/conda/feedstock_root/build_artifacts/truststore_1724770958874/work
urllib3 @ file:///home/conda/feedstock_root/build_artifacts/urllib3_1726496430923/work
zstandard==0.23.0
2024-10-10T16:11:26,356251874+00:00 | gunicorn/run |
2024-10-10T16:11:26,357903002+00:00 | gunicorn/run | Entry script directory: /var/azureml-app/scoring_scripts/.
2024-10-10T16:11:26,359612537+00:00 | gunicorn/run |
2024-10-10T16:11:26,361258564+00:00 | gunicorn/run | ###############################################
2024-10-10T16:11:26,362837681+00:00 | gunicorn/run | Dynamic Python Package Installation
2024-10-10T16:11:26,364359391+00:00 | gunicorn/run | ###############################################
2024-10-10T16:11:26,365602962+00:00 | gunicorn/run |
2024-10-10T16:11:26,367314098+00:00 | gunicorn/run | Dynamic Python package installation is disabled.
2024-10-10T16:11:26,368563070+00:00 | gunicorn/run |
2024-10-10T16:11:26,369805241+00:00 | gunicorn/run | ###############################################
2024-10-10T16:11:26,371040712+00:00 | gunicorn/run | Checking if the Python package azureml-inference-server-http is installed
2024-10-10T16:11:26,372294484+00:00 | gunicorn/run | ###############################################
2024-10-10T16:11:26,373504951+00:00 | gunicorn/run |
2024-10-10T16:11:26,580418857+00:00 | gunicorn/run | A required package azureml-inference-server-http is missing. Please install azureml-inference-server-http before trying again
2024-10-10T16:11:26,583849629+00:00 - gunicorn/finish 100 0
2024-10-10T16:11:26,585388641+00:00 - Exit code 100 is not normal. Killing image.
The problem being: A required package azureml-inference-server-http is missing. Please install azureml-inference-server-http before trying again
What am I doing wrong? What is the correct way to create and use environments, if I must build the image outside of Azure ML, and also must install the pip dependencies outside of Azure ML due to company restrictions? What are the alternatives?
Deployment description file:
$schema: https://azuremlschemas.azureedge.net/latest/managedOnlineDeployment.schema.json
model: azureml:model_name:model_version
code_configuration:
code: scoring_scripts
scoring_script: scorer.py
environment: azureml:env_name:env_version
endpoint_name: endpoint_name
instance_type: instance_type
instance_count: 1
Command:
az ml online-deployment create \
--name deployment_name \
--resource-group rg_name \
--workspace-name ws_name \
-f /path/top/file/online-deployment.yaml
Azure Machine Learning
-
Rohit Mungi 49,131 Reputation points Microsoft Employee Moderator2024-10-11T11:43:57.45+00:00 @Balla Péter Did you try to run an environment update to publish a new version with a minor change or create a new environment with a different name and check if it works? I think you might be encountering an issue with the environment cache while deploying since you have already built it with required dependencies. If the new environment or existing environment update with a new version generates a new hash value, then your subsequent deployment creation might go through.
If this still does not work, you might want to follow steps from the examples repo to use
az acr buildcommand to build a custom environment from a base image instead. See the sample deployment script here for a multi model or a single model here. Both these samples use custom models from a base model by specifying required pip packages in docker file and then use the environment in the deployment yaml file while runningaz ml online-deployment createI hope this helps!!
Sign in to comment
1 answer
Sort by: Most helpful
-
Balla Péter 5 Reputation points
2024-10-11T12:46:11.86+00:00 -
Rohit Mungi 49,131 Reputation points Microsoft Employee Moderator
2024-10-15T05:47:09.91+00:00 Thanks for update @Balla Péter
Sign in to comment -