R model deployment with custom Docker image: "ModuleNotFoundError: No module named 'azureml.api'"

Question

I am trying to deploy an R inference script to Azure ML Service Endpoint as an Azure Container Instance. I have made the following steps:

created a custom Docker image from scratch and pushed it to the Azure Container Registry (associated with AML Workspace)
registered a custom environment in AML Workspace, based on the image in ACR
deployed R entry script (just a simple hello world script with init() and run() functions defined)
- the inference configuration uses the custom AML environment
- deployment is made with Azure ML R SDK

The container instance is created, but the endpoint startup runs into error. Here is the output from the container instance:

2020-10-16T12:56:21,639812796+00:00 - gunicorn/run 
2020-10-16T12:56:21,639290594+00:00 - iot-server/run 
2020-10-16T12:56:21,640405198+00:00 - rsyslog/run 
2020-10-16T12:56:21,735291424+00:00 - nginx/run 
EdgeHubConnectionString and IOTEDGE_IOTHUBHOSTNAME are not set. Exiting...
2020-10-16T12:56:23,736657191+00:00 - iot-server/finish 1 0
2020-10-16T12:56:23,834747728+00:00 - Exit code 1 is normal. Not restarting iot-server.
Starting gunicorn 20.0.4
Listening at: http://127.0.0.1:31311 (11)
Using worker: sync
worker timeout is set to 300
Booting worker with pid: 38
/bin/bash: /root/miniconda3/lib/libtinfo.so.6: no version information available (required by /bin/bash)
SPARK_HOME not set. Skipping PySpark Initialization.
Exception in worker process
Traceback (most recent call last):
  File "/var/azureml-server/app.py", line 43, in 
    from azureml.api.exceptions.ClientSideException import ClientSideException
ModuleNotFoundError: No module named 'azureml.api'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/gunicorn/arbiter.py", line 583, in spawn_worker
    worker.init_process()
  File "/usr/lib/python3/dist-packages/gunicorn/workers/base.py", line 119, in init_process
    self.load_wsgi()
  File "/usr/lib/python3/dist-packages/gunicorn/workers/base.py", line 144, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/usr/lib/python3/dist-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/usr/lib/python3/dist-packages/gunicorn/app/wsgiapp.py", line 49, in load
    return self.load_wsgiapp()
  File "/usr/lib/python3/dist-packages/gunicorn/app/wsgiapp.py", line 39, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/usr/lib/python3/dist-packages/gunicorn/util.py", line 383, in import_app
    mod = importlib.import_module(module)
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "", line 1014, in _gcd_import
  File "", line 991, in _find_and_load
  File "", line 975, in _find_and_load_unlocked
  File "", line 671, in _load_unlocked
  File "", line 783, in exec_module
  File "", line 219, in _call_with_frames_removed
  File "/var/azureml-server/wsgi.py", line 1, in 
    import create_app
  File "/var/azureml-server/create_app.py", line 3, in 
    from app import main
  File "/var/azureml-server/app.py", line 45, in 
    from azure.ml.api.exceptions.ClientSideException import ClientSideException
ModuleNotFoundError: No module named 'azure.ml'
Worker exiting (pid: 38)
Shutting down: Master
Reason: Worker failed to boot.
2020-10-16T12:56:39,434787859+00:00 - gunicorn/finish 3 0
2020-10-16T12:56:39,435715063+00:00 - Exit code 3 is not normal. Killing image.

How do I install the azureml.api dependency, which can not be found? It doesn't seem to be part of the Azure ML SDK. I have installed the following dependencies in my Dockerfile:

RUN apt-get -y install python3-flask python3-rpy2 python3-azure python3-applicationinsights
RUN pip install azureml-core

I also have Miniconda installed. Pip refers to Miniconda's pip.

Or, is this dependency available to install at all? Should I use some pre-defined AML environment as the base Docker image? (Note: I am currently using bare FROM: ubuntu). Suggestions how to find and use the base images are also welcome, since this is not documented very well.

Accepted Answer

@Lauri Lehman Thanks for the question. Here is the document to Create Custom Docker Base Images for Azure Machine Learning Environments for R people.
We have used the AzureML RScriptStep pipeline feature which allows you to point to CRAN or Github or custom URLS, but this requires authoring the pipeline in python or YAML.. In R You can also these arguments in the Azuremlsdk R estimator function: https://azure.github.io/azureml-sdk-for-r/reference/estimator.html
Another option that are not available through conda install as part of the R script with install.packages(“path/*.tar.gz”, repos=NULL))

One of the challenges is that the build at runtime can take a while to prepare the environment. R likes to compile packages on Linux environments and a large package could have lots of dependencies which would take a while. This is an R on Linux/PaaS thing, rather than specific to AzureML

To make start up fast we created a custom docker image where you can tightly control the image ahead of runtime. If you want to go in this direction you can find an example Dockerfile to get you started here..
https://github.com/Azure/azureml-sdk-for-r/blob/master/.azure-pipelines/docker/Dockerfile

R model deployment with custom Docker image: "ModuleNotFoundError: No module named 'azureml.api'"

0 additional answers