How to configure the runs combine pipeline and hyperdrive

Question

Hi there,

I have a pipeline that is composed of "data prep" and "training" steps and I need to test different hyperparameters on the training step. I cannot figure out how I should use hyperdrive config within the pipeline.

from azureml.core import Environment
from azureml.core.conda_dependencies import CondaDependencies
from azureml.core.runconfig import RunConfiguration

Create a Python environment for the experiment

diabetes_env = Environment("diabetes-pipeline-env")
diabetes_env.python.user_managed_dependencies = False # Let Azure ML manage dependencies
diabetes_env.docker.enabled = True # Use a docker container

Create a set of package dependencies

diabetes_packages = CondaDependencies.create(conda_packages=['scikit-learn','ipykernel','matplotlib','pandas','pip'],
pip_packages=['azureml-defaults','azureml-dataprep[pandas]','pyarrow'])

Add the dependencies to the environment

diabetes_env.python.conda_dependencies = diabetes_packages

Register the environment

diabetes_env.register(workspace=ws)
registered_env = Environment.get(ws, 'diabetes-pipeline-env')

Create a new runconfig object for the pipeline

run_config = RunConfiguration()

Use the compute you created above.

run_config.target = ComputerTarget_Crea

Assign the environment to the run configuration

run_config.environment = registered_env

print ("Run configuration created.")

Get the training dataset

diabetes_ds = ws.datasets.get("diabetes dataset")

Create a PipelineData (temporary Data Reference) for the model folder

prepped_data_folder = PipelineData("prepped_data_folder", datastore=ws.get_default_datastore())

Create a script config

script_config = ScriptRunConfig(source_directory=experiment_folder,
script='diabetes_training.py',
# Add non-hyperparameter arguments -in this case, the training dataset
arguments = ['--input-data', diabetes_ds.as_named_input('training_data')],
environment=registered_env,
compute_target = ComputerTarget_Crea)

Sample a range of parameter values

params = GridParameterSampling(
{
# Hyperdrive will try 6 combinations, adding these as script arguments
'--learning_rate': choice(0.01, 0.1, 1.0),
'--n_estimators' : choice(10, 100)
}
)

hyperdrive = HyperDriveConfig(run_config=script_config,
hyperparameter_sampling=params,
policy=None, # No early stopping policy
primary_metric_name='AUC', # Find the highest AUC metric
primary_metric_goal=PrimaryMetricGoal.MAXIMIZE,
max_total_runs=6, # Restict the experiment to 6 iterations
max_concurrent_runs=2) # Run up to 2 iterations in parallel

Step 1, Run the data prep script

prep_step = PythonScriptStep(name = "Prepare Data",
source_directory = experiment_folder,
script_name = "prep_diabetes.py",
arguments = ['--input-data', diabetes_ds.as_named_input('raw_data'),
'--prepped-data', prepped_data_folder],
outputs=[prepped_data_folder],
compute_target = ComputerTarget_Crea,
runconfig = run_config,
allow_reuse = True)

Step 2, run the training script

train_step = PythonScriptStep(name = "Train and Register Model",
source_directory = experiment_folder,
script_name = "train_diabetes.py",
arguments = ['--training-folder', prepped_data_folder],
inputs=[prepped_data_folder],
compute_target = ComputerTarget_Crea,
runconfig = hyperdrive,
allow_reuse = True)

Error I get

Expected a StepRun object but received <class 'azureml.core.run.Run'> instead.
This usually indicates a package conflict with one of the dependencies of azureml-core or azureml-pipeline-core.
Please check for package conflicts in your python environment

TypeError Traceback (most recent call last)
~\AppData\Roaming\Python\Python38\site-packages\azureml\pipeline\core\run.py in wait_for_completion(self, show_output, timeout_seconds, raise_on_error)
293 try:
--> 294 step_run.wait_for_completion(timeout_seconds=timeout_seconds - time_elapsed,
295 raise_on_error=raise_on_error)

TypeError: wait_for_completion() got an unexpected keyword argument 'timeout_seconds'

During handling of the above exception, another exception occurred:

ActivityFailedException Traceback (most recent call last)
<ipython-input-29-e1c5033d6731> in <module>
13 print("Pipeline submitted for execution.")
14 RunDetails(pipeline_run).show()
---> 15 pipeline_run.wait_for_completion(show_output=True)

~\AppData\Roaming\Python\Python38\site-packages\azureml\pipeline\core\run.py in wait_for_completion(self, show_output, timeout_seconds, raise_on_error)
307 'azureml-core or azureml-pipeline-core.\n' +
308 'Please check for package conflicts in your python environment')
--> 309 step_run.wait_for_completion(raise_on_error=raise_on_error)
310 else:
311 # Different error than the run rehydration issue

~\anaconda3\lib\site-packages\azureml\core\run.py in wait_for_completion(self, show_output, wait_post_processing, raise_on_error)
819
820 if raise_on_error:
--> 821 raise ActivityFailedException(error_details=json.dumps(error, indent=4))
822
823 return final_details

ActivityFailedException: ActivityFailedException:
Message: Activity Failed:
{
"error": {
"code": "UserError",
"message": "Unexpected character encountered while parsing value: <. Path '', line 0, position 0.",
"messageParameters": {},
"details": []
},
"time": "0001-01-01T00:00:00.000Z"
}
InnerException None
ErrorResponse
{
"error": {
"message": "Activity Failed:\n{\n \"error\": {\n \"code\": \"UserError\",\n \"message\": \"Unexpected character encountered while parsing value: <. Path '', line 0, position 0.\",\n \"messageParameters\": {},\n \"details\": []\n },\n \"time\": \"0001-01-01T00:00:00.000Z\"\n}"
}
}

Answer 1

@Mohsen Sichani Thanks for the question. Please follow the sample that demonstrate the use of HyperDriveStep in AML Pipeline.
https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-parameter-tuning-with-hyperdrive.ipynb