azure machine learning - environment - add conda dependancies by sdk for training on a compute instance

nabil ouddane 11

hello guys, i m a bit confused

i need to add joblib package to a tensorflow 2.7 environment so as to load the needed datas during the train on the compute instance

1) i start by making an environnment clone:

env = Environment.get(workspace=ws, name="AzureML-tensorflow-2.7-ubuntu20.04-py38-cuda11-gpu")
curated_clone = env.clone("customize_curated")

2) i add conda dependancies at the python key (like the MS example)

conda_dep = CondaDependencies()
conda_dep.add_pip_package("joblib==1.0.1")
curated_clone.python.conda_dependencies=conda_dep

3) registration

curated_clone.register(workspace=ws)

i meet an error when i train the model by using the registred environment caus joblib is missing

but if i modify the dockerfile of the environmment by adding joblib directly on studio, everything is ok
could you please explain me what i m doing wrong with the sdk

YutongTie-MSFT 51,741

Hello @nabil ouddane

Could you please share the error message and which guidance you are following to us? I just went through the Python SDK document and below sample works to me.

from azureml.core.environment import Environment  
from azureml.core.conda_dependencies import CondaDependencies  
  
myenv = Environment(name="myenv")  
conda_dep = CondaDependencies()  
  
# Installs numpy version 1.17.0 conda package  
conda_dep.add_conda_package("numpy==1.17.0")  
  
# Installs pillow package  
conda_dep.add_pip_package("pillow")  
conda_dep.add_pip_package("joblib==1.0.1")  
  
# Adds dependencies to PythonSection of myenv  
myenv.python.conda_dependencies=conda_dep

I hope this works to you too. If this is not working, please share the guidance you are following so that I can reproduce.

Regards,
Yutong

nabil ouddane 11

import os, shutil
import pandas as pd
from joblib import dump, load

Create a folder for the experiment files

training_folder = 'test-dir'
os.makedirs(training_folder, exist_ok=True)

dump test data joblib

data = {'product_name': ['laptop', 'printer', 'tablet', 'desk', 'chair'],
        'price': [1200, 150, 300, 450, 200]
        }
df = pd.DataFrame(data)
dump(df,os.path.join(training_folder, 'test.joblib'))

write script

%%writefile $training_folder/test.py

import os
from azureml.core import Run
# Get the experiment run context
run = Run.get_context()

#load data
from joblib import dump, load

data = load('test.joblib', mmap_mode='r')
run.complete()

nabil ouddane 11

submit the experiment run

from azureml.core import Experiment, ScriptRunConfig, Environment, Workspace
from azureml.core.compute import ComputeTarget
from azureml.core.environment import Environment
from azureml.core.conda_dependencies import CondaDependencies

ws = Workspace.from_config()
compute_name = 'P7-cpu' #'Standard_DS3_v2'

env = Environment.get(workspace=ws, name="AzureML-tensorflow-2.7-ubuntu20.04-py38-cuda11-gpu")
env_clone = env.clone("custom_env")
# Adds dependencies
conda_dep = CondaDependencies()
conda_dep.add_pip_package("joblib==1.0.1")
env_clone.python.conda_dependencies=conda_dep

script_config = ScriptRunConfig(source_directory=training_folder,
                      script='test.py',
                      compute_target=ComputeTarget(workspace=ws, name=compute_name),
                      environment=env_clone)

experiment_name = 'test-train'
experiment = Experiment(workspace=ws, name=experiment_name)
run = experiment.submit(config=script_config)

run.wait_for_completion()

nabil ouddane 11 Reputation points

2022-02-24T09:29:20.02+00:00

here is the error:
[2022-02-24T08:53:35.895570] The experiment failed. Finalizing run...
Cleaning up all outstanding Run operations, waiting 900.0 seconds
2 items cleaning up...
Cleanup took 0.10983729362487793 seconds
Traceback (most recent call last):
File "test.py", line 8, in <module>
from joblib import dump, load
ModuleNotFoundError: No module named 'joblib'
YutongTie-MSFT 51,741 Reputation points

2022-02-24T16:12:17.287+00:00

@nabil ouddane

May I know are you working on Jupyter, ML studio Notebook, or VS so that I can reproduce? Thanks.
nabil ouddane 11 Reputation points

2022-02-24T17:45:37.187+00:00

jupyter lab on my computer.
thank you for your help
YutongTie-MSFT 51,741 Reputation points

2022-02-25T00:19:58.597+00:00

@nabil ouddane

I think I got your error, I can reproduce the issue now. This issue may be caused by packages versions incompatible. I have forwarded this issue for investigation and fixing. Thanks for reporting this to us.

Regards,
Yutong
nabil ouddane 11 Reputation points

2022-02-25T08:58:15.08+00:00

i tested several packages versions but same error.
i tested with the example of pillow, same error
conda dependancies seem not to be readed
do you think you will get an answer soon?
YutongTie-MSFT 51,741 Reputation points

2022-03-01T07:32:15.847+00:00

Hi @nabil ouddane

Just a double check , are you willing to do the support ticket? Please feel free to let me know. Thanks.

Regards,
Yutong

Share via

azure machine learning - environment - add conda dependancies by sdk for training on a compute instance

Create a folder for the experiment files

dump test data joblib

write script

submit the experiment run

Your answer