Hyperparameter search with HyperDriveConfig for ReinforcementLearningEstimator

Question

Hello,

I would like to automatize Hyperparameter search for my Reinforcement Learning training jobs.
In my understanding the workflow can be written as

training_estimator = rl.ReinforcementLearningEstimator(
        source_directory='./src',
        entry_script=entry_script,
        script_params=script_params,
        compute_target=compute_target,
        rl_framework=rl.Ray(),
        environment=env_settings)

    grid_sampling = hyperdrive.GridParameterSampling(parameter_space={'lr': hyperdrive.choice([0.007, 0.005])})

    hd_config = hyperdrive.HyperDriveConfig(estimator=training_estimator,
                                            hyperparameter_sampling=grid_sampling,
                                            policy=early_termination_policy,
                                            primary_metric_name="episode_reward_mean",
                                            primary_metric_goal=hyperdrive.PrimaryMetricGoal.MAXIMIZE,
                                            max_total_runs=100,
                                            max_concurrent_runs=4)

    run = exp.submit(hd_config)

However, this gets rejected because ReinforcementLearningEstimator does not implement MMLBaseEstimator.

Is there any trick that I'm missing or this use case isn't supported by Azure?

EDIT:

This code fails with the following error message

Traceback (most recent call last):
  File "hyperdriverun.py", line 188, in <module>
    run = exp.submit(hd_config)
  File "/home/.../python3.7/site-packages/azureml/core/experiment.py", line 220, in submit
    run = submit_func(config, self.workspace, self.name, **kwargs)
  File "/home/.../python3.7/site-packages/azureml/train/hyperdrive/_search.py", line 145, in search
    telemetry_values, activity_logger, **kwargs)
  File "/home/.../python3.7/site-packages/azureml/train/hyperdrive/_search.py", line 38, in _create_experiment_dto
    platform_config = hyperdrive_config._get_platform_config(workspace, experiment_name, **kwargs)
  File "/home/.../python3.7/site-packages/azureml/train/hyperdrive/runconfig.py", line 672, in _get_platform_config
    platform_config.update(self._get_platform_config_data_from_run_config(workspace))
  File "/home/.../python3.7/site-packages/azureml/train/hyperdrive/runconfig.py", line 684, in _get_platform_config_data_from_run_config
    run_config = get_run_config_from_script_run(self.estimator._get_script_run_config())
AttributeError: 'ReinforcementLearningEstimator' object has no attribute '_get_script_run_config'

Answer 1

Hi, thanks for the information. From what I understand, you're trying to use the hyperdrive config along with reinforcement learning estimator. Hyperdrive uses scriptrunconfig which this is missing in the submit, so the error is seen. Ideally, hyperdrive will use scriptrunconfig class. Here's an example, hyperdrive config uses run_config as src which in turn is scriptconfig class. The following document explains how to use this estimator in an experiment. Hope this helps.