question

DavidClarance-7033 avatar image
0 Votes"
DavidClarance-7033 asked ramr-msft commented

Unable to activate conda enviornment in AzureML when deploying pipeline with custom docker container

I have trouble using the environment I've created in a custom docker container while deploying an AzureML pipeline.

Here is what I am doing:

  1. I create a container which builds a conda enviornment along with other requirements.

  2. The above container is create on Azure Pipelines and stored on a private azure container repository.

  3. I pull this container in my pipeline using the below:

aml_run_config.environment.docker.base_image = "xxxx"
    aml_run_config.environment.docker.base_image_registry.address = (
        "xxxx.azurecr.io"
    )
    aml_run_config.environment.docker.base_image_registry.username = (
        "xxxx"
    )
    aml_run_config.environment.docker.base_image_registry.password = (
        xxxx
    )


After that I tried two things:

  1. Use a PythonScriptStep() to call my forecast.py

  2. Use a CommandStep() with command = "conda run -n build_env python src/forecast.py"

build_env is the name of my conda env

In both cases I get a ModuleNotFoundError for pandas (the first library I use).

If I pull my docker container locally and try exactly the same command, it works fine.

What am I missing?








azure-machine-learningazure-machine-learning-inference
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@DavidClarance-7033 Thanks, Could you please share the error logs.

0 Votes 0 ·

1 Answer

ramr-msft avatar image
0 Votes"
ramr-msft answered ramr-msft commented

@DavidClarance-7033 Thanks for the question. pre-built Azure ML environments: https://github.com/Azure/AzureML-Containers#base-image-dependencies

It is possible to use our own docker image (which is pushed into private container registry!)

Deploy models with custom Docker image - Azure Machine Learning | Microsoft Docs

To use an image from a private container registry that is not in your workspace, you must use docker.base_image_registry to specify the address of the repository and a user name and password:



   # Set the container registry information
     myenv.docker.base_image_registry.address = "myregistry.azurecr.io"
     myenv.docker.base_image_registry.username = "username"
     myenv.docker.base_image_registry.password = "password"
        
     myenv.inferencing_stack_version = "latest"  # This will install the inference specific apt packages.
        
     # Define the packages needed by the model and scripts
     from azureml.core.conda_dependencies import CondaDependencies
     conda_dep = CondaDependencies()
     # you must list azureml-defaults as a pip dependency
     conda_dep.add_pip_package("azureml-defaults")
     myenv.python.conda_dependencies=conda_dep




· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thanks @ramr-msft. I tried add the statements as you suggest but I still get the same error. I realize I'm adding packages at different stages so maybe that's the issue.

  1. I add packages in the docker container.

  2. I add a private package in the azure pipeline that builds the docker container using the Docker@2 task.

  3. I add azureml-defaults (as per your recommendation) in the CondaDependencies() workflow.

Here's the error message:

 {
     "error": {
         "code": "UserError",
         "message": "User program failed with ModuleNotFoundError: No module named 'pandas'",
         "messageParameters": {},
         "detailsUri": "https://aka.ms/azureml-run-troubleshooting",
         "details": []
     },
     "time": "0001-01-01T00:00:00.000Z"
 }
         InnerException None
         ErrorResponse
 {
     "error": {
         "message": "Activity Failed:\n{\n    \"error\": {\n        \"code\": \"UserError\",\n        \"message\": \"User program failed with ModuleNotFoundError: No module named 'pandas'\",\n
     \"messageParameters\": {},\n        \"detailsUri\": \"https://aka.ms/azureml-run-troubleshooting\",\n        \"details\": []\n    },\n    \"time\": \"0001-01-01T00:00:00.000Z\"\n}"
     }
 }



0 Votes 0 ·

@DavidClarance-7033 Thanks for the details. Here is the troubleshooting doc for module not found error that could help to debug.
https://docs.microsoft.com/en-us/azure/machine-learning/how-to-set-up-training-targets#troubleshooting

0 Votes 0 ·