Edit

Hyperparameter tuning a model (v2)

APPLIES TO: Azure CLI ml extension v2 (current) Python SDK azure-ai-ml v2 (current)

In this article, you learn how to automate efficient hyperparameter tuning with the Azure Machine Learning SDK v2 and CLI v2 by using the SweepJob class.

  • Define the parameter search space
  • Choose a sampling algorithm
  • Set the optimization objective
  • Configure an early termination policy
  • Set sweep job limits
  • Submit the experiment
  • Visualize training jobs
  • Select the best configuration

Before you begin

Before you run a sweep job, ensure you have:

  • An Azure Machine Learning workspace and a compute target such as a CPU cluster.
  • A training script that logs the objective metric with the exact name used in primary_metric.
  • The Azure ML SDK v2 installed in your Python environment, and authenticated with DefaultAzureCredential or az login.
  • The Azure CLI and Azure ML CLI extension installed if you use the YAML-based examples in this article.

The examples in this article assume an existing workspace and a compute target that the current user can access.

What is hyperparameter tuning?

Hyperparameters are adjustable settings that control model training. For neural networks, you choose the number of hidden layers and the number of nodes per layer. Model performance depends heavily on these values.

Hyperparameter tuning (or hyperparameter optimization) is the process of finding the hyperparameter configuration that yields the best performance. This process is often computationally expensive and manual.

Azure Machine Learning automates hyperparameter tuning and runs experiments in parallel to efficiently optimize hyperparameters.

Define the search space

Tune hyperparameters by exploring the range of values defined for each hyperparameter.

Hyperparameters can be discrete or continuous, and can have a value distribution expressed with a parameter expression.

Discrete hyperparameters

Specify discrete hyperparameters as a Choice among discrete values. Choice can be:

  • one or more comma-separated values
  • a range object
  • any arbitrary list object
from azure.ai.ml.sweep import Choice

command_job_for_sweep = command_job(
    batch_size=Choice(values=[16, 32, 64, 128]),
    number_of_hidden_layers=Choice(values=range(1,5)),
)

References:

In this case, batch_size takes one of [16, 32, 64, 128] and number_of_hidden_layers takes one of [1, 2, 3, 4].

You can also specify the following advanced discrete hyperparameters by using a distribution:

  • QUniform(min_value, max_value, q) - Returns a value like round(Uniform(min_value, max_value) / q) * q.
  • QLogUniform(min_value, max_value, q) - Returns a value like round(exp(Uniform(min_value, max_value)) / q) * q.
  • QNormal(mu, sigma, q) - Returns a value like round(Normal(mu, sigma) / q) * q.
  • QLogNormal(mu, sigma, q) - Returns a value like round(exp(Normal(mu, sigma)) / q) * q.

Continuous hyperparameters

Specify continuous hyperparameters as a distribution over a continuous range of values:

  • Uniform(min_value, max_value) - Returns a value uniformly distributed between min_value and max_value.
  • LogUniform(min_value, max_value) - Returns a value drawn according to exp(Uniform(min_value, max_value)) so that the logarithm of the return value is uniformly distributed.
  • Normal(mu, sigma) - Returns a real value that's normally distributed with mean mu and standard deviation sigma.
  • LogNormal(mu, sigma) - Returns a value drawn according to exp(Normal(mu, sigma)) so that the logarithm of the return value is normally distributed.

An example of a parameter space definition:

from azure.ai.ml.sweep import Normal, Uniform

command_job_for_sweep = command_job(   
    learning_rate=Normal(mu=10, sigma=3),
    keep_probability=Uniform(min_value=0.05, max_value=0.1),
)

References:

This code defines a search space with two parameters - learning_rate and keep_probability. learning_rate has a normal distribution with mean value 10 and a standard deviation of 3. keep_probability has a uniform distribution with a minimum value of 0.05 and a maximum value of 0.1.

For the CLI, use the sweep job YAML schema to define the search space:

    search_space:
        conv_size:
            type: choice
            values: [2, 5, 7]
        dropout_rate:
            type: uniform
            min_value: 0.1
            max_value: 0.2

Sampling the hyperparameter space

Specify the sampling method for the hyperparameter space. Azure Machine Learning supports:

  • Random sampling
  • Grid sampling
  • Bayesian sampling

Random sampling

Random sampling supports discrete and continuous hyperparameters, and supports early termination of low-performing jobs. Many users start with random sampling to identify promising regions, then refine.

In random sampling, values are drawn uniformly (or via the specified random rule) from the defined search space. After creating your command job, use sweep to define the sampling algorithm.

from azure.ai.ml.entities import CommandJob
from azure.ai.ml.sweep import RandomSamplingAlgorithm, SweepJob, SweepJobLimits

   command_job = CommandJob(
       inputs=dict(kernel="linear", penalty=1.0),
       compute=cpu_cluster,
       environment=f"{job_env.name}:{job_env.version}",
       code="./scripts",
       command="python scripts/train.py --kernel $kernel --penalty $penalty",
       experiment_name="sklearn-iris-flowers",
   )

   sweep = SweepJob(
       sampling_algorithm=RandomSamplingAlgorithm(seed=999, rule="sobol", logbase="e"),
       trial=command_job,
       search_space={"ss": Choice(type="choice", values=[{"space1": True}, {"space2": True}])},
       inputs={"input1": {"file": "top_level.csv", "mode": "ro_mount"}},  # type:ignore
       compute="top_level",
       limits=SweepJobLimits(trial_timeout=600),
   )

References:

Sobol

Sobol is a quasi-random sequence that improves space-filling and reproducibility. Provide a seed and set rule="sobol" on RandomSamplingAlgorithm.

from azure.ai.ml.sweep import  RandomSamplingAlgorithm

sweep_job = command_job_for_sweep.sweep(
    compute="cpu-cluster",
    sampling_algorithm = RandomSamplingAlgorithm(seed=123, rule="sobol"),
    ...
)

References: RandomSamplingAlgorithm

Grid sampling

Grid sampling supports discrete hyperparameters. Use grid sampling if you can budget to exhaustively search over the search space. Supports early termination of low-performance jobs.

Grid sampling does a simple grid search over all possible values. Grid sampling can only be used with choice hyperparameters. For example, the following space has six samples:

from azure.ai.ml.sweep import Choice

command_job_for_sweep = command_job(
    batch_size=Choice(values=[16, 32]),
    number_of_hidden_layers=Choice(values=[1,2,3]),
)

sweep_job = command_job_for_sweep.sweep(
    compute="cpu-cluster",
    sampling_algorithm = "grid",
    ...
)

References: Choice

Bayesian sampling

Bayesian sampling (Bayesian optimization) selects new samples based on prior results to improve the primary metric efficiently.

Use Bayesian sampling if you have enough budget to explore the hyperparameter space. For best results, set the maximum number of jobs to at least 20 times the number of hyperparameters you're tuning.

The number of concurrent jobs affects how effective the tuning process is. A smaller number of concurrent jobs might lead to better sampling convergence. The smaller degree of parallelism increases the number of jobs that benefit from previously completed jobs.

Bayesian sampling supports choice, uniform, and quniform distributions.

from azure.ai.ml.sweep import Uniform, Choice

command_job_for_sweep = command_job(   
    learning_rate=Uniform(min_value=0.05, max_value=0.1),
    batch_size=Choice(values=[16, 32, 64, 128]),
)

sweep_job = command_job_for_sweep.sweep(
    compute="cpu-cluster",
    sampling_algorithm = "bayesian",
    ...
)

References:

Specify the objective of the sweep

Define the objective of your sweep job by specifying the primary metric and goal you want hyperparameter tuning to optimize. Each training job is evaluated for the primary metric. The early termination policy uses the primary metric to identify low-performance jobs.

  • primary_metric: The name of the primary metric needs to exactly match the name of the metric logged by the training script.
  • goal: It can be either maximize or minimize and determines whether the primary metric is maximized or minimized when evaluating the jobs.
from azure.ai.ml.sweep import Uniform, Choice

command_job_for_sweep = command_job(   
    learning_rate=Uniform(min_value=0.05, max_value=0.1),
    batch_size=Choice(values=[16, 32, 64, 128]),
)

sweep_job = command_job_for_sweep.sweep(
    compute="cpu-cluster",
    sampling_algorithm = "bayesian",
    primary_metric="accuracy",
    goal="maximize",
)

References:

This sample maximizes "accuracy".

Log metrics for hyperparameter tuning

Your training script must log the primary metric with the exact name expected by the sweep job.

Log the primary metric in your training script by using the following sample snippet:

import mlflow
mlflow.log_metric("accuracy", float(val_accuracy))

References: mlflow.log_metric

The training script calculates the val_accuracy and logs it as the primary metric "accuracy". Each time the metric is logged, the hyperparameter tuning service receives it. You decide how often to report the metric.

For more information about logging values for training jobs, see Enable logging in Azure Machine Learning training jobs.

Specify early termination policy

End poorly performing jobs early to improve efficiency.

You can configure the following parameters that control when a policy is applied:

  • evaluation_interval: the frequency of applying the policy. Each time the training script logs the primary metric counts as one interval. An evaluation_interval of 1 applies the policy every time the training script reports the primary metric. An evaluation_interval of 2 applies the policy every other time.
  • delay_evaluation: delays the first policy evaluation for a specified number of intervals. This optional parameter avoids premature termination of training jobs by allowing all configurations to run for a minimum number of intervals. If specified, the policy applies every multiple of evaluation_interval that is greater than or equal to delay_evaluation. If not specified, delay_evaluation defaults to 0.

The default evaluation_interval value depends on the early termination policy you choose.

Azure Machine Learning supports the following early termination policies:

Bandit policy

The Bandit policy uses a slack factor or slack amount plus an evaluation interval. It ends a job when its primary metric falls outside the allowed slack from the best job.

Specify the following configuration parameters:

  • slack_factor or slack_amount: Allowed difference from the best job. slack_factor is a ratio; slack_amount is an absolute value.

    For example, consider a Bandit policy applied at interval 10. Assume that the best performing job at interval 10 reported a primary metric of 0.8 with a goal to maximize the primary metric. If the policy specifies a slack_factor of 0.2, the policy terminates any training jobs whose best metric at interval 10 is less than 0.66 (0.8/(1+slack_factor)).

  • evaluation_interval: (optional) the frequency for applying the policy

  • delay_evaluation: (optional) delays the first policy evaluation for a specified number of intervals

from azure.ai.ml.sweep import BanditPolicy
sweep_job.early_termination = BanditPolicy(slack_factor = 0.1, delay_evaluation = 5, evaluation_interval = 1)

References: BanditPolicy

In this example, the early termination policy is applied at every interval when metrics are reported, starting at evaluation interval 5. The policy terminates any jobs whose best metric is less than (1/(1+0.1)) or 91% of the best performing jobs.

Median stopping policy

The median stopping policy is an early termination policy that uses running averages of primary metrics reported by the jobs. This policy computes running averages across all training jobs and stops jobs whose primary metric value is worse than the median of the averages.

This policy takes the following configuration parameters:

  • evaluation_interval: how often to apply the policy (optional).
  • delay_evaluation: number of intervals to delay the first policy evaluation (optional).
from azure.ai.ml.sweep import MedianStoppingPolicy
sweep_job.early_termination = MedianStoppingPolicy(delay_evaluation = 5, evaluation_interval = 1)

References: MedianStoppingPolicy

In this example, the early termination policy is applied at every interval starting at evaluation interval 5. A job stops at interval 5 if its best primary metric is worse than the median of the running averages over intervals 1 through 5 across all training jobs.

Truncation selection policy

The truncation selection policy cancels a percentage of the lowest performing jobs at each evaluation interval. Jobs are compared by using the primary metric.

This policy takes the following configuration parameters:

  • truncation_percentage: the percentage of lowest performing jobs to terminate at each evaluation interval. An integer value between 1 and 99.
  • evaluation_interval: how often to apply the policy (optional).
  • delay_evaluation: number of intervals to delay the first policy evaluation (optional).
from azure.ai.ml.sweep import TruncationSelectionPolicy
sweep_job.early_termination = TruncationSelectionPolicy(
    evaluation_interval=1,
    truncation_percentage=20,
    delay_evaluation=5,
)

References: TruncationSelectionPolicy

In this example, the early termination policy is applied at every interval starting at evaluation interval 5. A job terminates at interval 5 if its performance at interval 5 is in the lowest 20% of performance of all jobs at interval 5.

No termination policy (default)

If you don't specify a policy, the hyperparameter tuning service lets all training jobs run to completion.

sweep_job.early_termination = None

References: SweepJob

Choosing an early termination policy

  • For a conservative policy that saves resources without terminating promising jobs, consider a Median Stopping Policy with evaluation_interval set to 1 and delay_evaluation set to 5. These settings are conservative and can provide about 25%-35% savings with no loss on the primary metric, based on evaluation data.
  • For more aggressive savings, use a Bandit Policy with a smaller allowable slack or a Truncation Selection Policy with a larger truncation percentage.

Set limits for your sweep job

Control your resource budget by setting limits for your sweep job.

  • max_total_trials: Maximum number of trial jobs. Must be an integer between 1 and 1000.
  • max_concurrent_trials: (optional) Maximum number of trial jobs that can run concurrently. If not specified, max_total_trials number of jobs launch in parallel. If specified, must be an integer between 1 and 1000.
  • timeout: Maximum time in seconds the entire sweep job is allowed to run. Once this limit is reached the system cancels the sweep job, including all its trials.
  • trial_timeout: Maximum time in seconds each trial job is allowed to run. Once this limit is reached the system cancels the trial.

Note

If both max_total_trials and timeout are specified, the hyperparameter tuning experiment terminates when the first of these two thresholds is reached.

Note

The number of concurrent trial jobs is gated on the resources available in the specified compute target. Ensure that the compute target has the available resources for the desired concurrency.

sweep_job.set_limits(max_total_trials=20, max_concurrent_trials=4, timeout=1200)

References: SweepJob.set_limits

This code configures the hyperparameter tuning experiment to use a maximum of 20 total trial jobs, running four trial jobs at a time with a timeout of 1,200 seconds for the entire sweep job.

Configure hyperparameter tuning experiment

To configure your hyperparameter tuning experiment, provide the following information:

  • The defined hyperparameter search space
  • Your sampling algorithm
  • Your early termination policy
  • Your objective
  • Resource limits
  • CommandJob or CommandComponent
  • SweepJob

SweepJob can run a hyperparameter sweep on the Command or Command Component.

Note

The compute target used in sweep_job must have enough resources to satisfy your concurrency level. For more information about compute targets, see Compute targets.

Configure your hyperparameter tuning experiment:

from azure.ai.ml import MLClient
from azure.ai.ml import command, Input
from azure.ai.ml.sweep import Choice, Uniform, MedianStoppingPolicy
from azure.identity import DefaultAzureCredential

# Create your base command job
command_job = command(
    code="./src",
    command="python main.py --iris-csv ${{inputs.iris_csv}} --learning-rate ${{inputs.learning_rate}} --boosting ${{inputs.boosting}}",
    environment="AzureML-lightgbm-3.2-ubuntu18.04-py37-cpu@latest",
    inputs={
        "iris_csv": Input(
            type="uri_file",
            path="https://azuremlexamples.blob.core.windows.net/datasets/iris.csv",
        ),
        "learning_rate": 0.9,
        "boosting": "gbdt",
    },
    compute="cpu-cluster",
)

# Override your inputs with parameter expressions
command_job_for_sweep = command_job(
    learning_rate=Uniform(min_value=0.01, max_value=0.9),
    boosting=Choice(values=["gbdt", "dart"]),
)

# Call sweep() on your command job to sweep over your parameter expressions
sweep_job = command_job_for_sweep.sweep(
    compute="cpu-cluster",
    sampling_algorithm="random",
    primary_metric="test-multi_logloss",
    goal="minimize",
)

# Specify your experiment details
sweep_job.display_name = "lightgbm-iris-sweep-example"
sweep_job.experiment_name = "lightgbm-iris-sweep-example"
sweep_job.description = "Run a hyperparameter sweep job for LightGBM on Iris dataset."

# Define the limits for this sweep
sweep_job.set_limits(max_total_trials=20, max_concurrent_trials=10, timeout=7200)

# Set early stopping on this one
sweep_job.early_termination = MedianStoppingPolicy(
    delay_evaluation=5, evaluation_interval=2
)

References:

The command_job is invoked as a function so you can apply parameter expressions. Configure the sweep function with trial, sampling algorithm, objective, limits, and compute. The following snippet comes from the sample notebook Run hyperparameter sweep on a Command or CommandComponent. In this sample, you tune learning_rate and boosting. The MedianStoppingPolicy drives early stopping. This policy stops a job whose primary metric is worse than the median of running averages across all jobs. For more information, see MedianStoppingPolicy reference.

To see how the parameter values are received, parsed, and passed to the training script for tuning, refer to this code sample.

Important

Every hyperparameter sweep job restarts the training from scratch, including rebuilding the model and all the data loaders. You can minimize this cost by using an Azure Machine Learning pipeline or manual process to do as much data preparation as possible before your training jobs.

Submit hyperparameter tuning experiment

After you define your hyperparameter tuning configuration, submit the job:

# submit the sweep
returned_sweep_job = ml_client.create_or_update(sweep_job)
# get a URL for the status of the job
returned_sweep_job.services["Studio"].endpoint

References:

Visualize hyperparameter tuning jobs

Visualize hyperparameter tuning jobs in Azure Machine Learning studio. For details, see View job records in the studio.

  • Metrics chart: This visualization tracks the metrics logged for each hyperdrive child job over the duration of hyperparameter tuning. Each line represents a child job, and each point measures the primary metric value at that iteration of runtime.

    Hyperparameter tuning metrics chart

  • Parallel Coordinates Chart: This visualization shows the correlation between primary metric performance and individual hyperparameter values. The chart is interactive via movement of axes (select and drag by the axis label), and by highlighting values across a single axis (select and drag vertically along a single axis to highlight a range of desired values). The parallel coordinates chart includes an axis on the rightmost portion of the chart that plots the best metric value corresponding to the hyperparameters set for that job instance. This axis is provided in order to project the chart gradient legend onto the data in a more readable fashion.

    Hyperparameter tuning parallel coordinates chart

  • 2-Dimensional Scatter Chart: This visualization shows the correlation between any two individual hyperparameters along with their associated primary metric value.

    Hyparameter tuning 2-dimensional scatter chart

  • 3-Dimensional Scatter Chart: This visualization is the same as 2D but allows for three hyperparameter dimensions of correlation with the primary metric value. You can also select and drag to reorient the chart to view different correlations in 3D space.

    Hyparameter tuning 3-dimensional scatter chart

Find the best trial job

When all tuning jobs finish, retrieve the best trial outputs:

# Download best trial model output
ml_client.jobs.download(returned_sweep_job.name, output_name="model")

References:

Use the CLI to download all default and named outputs of the best trial job and logs of the sweep job.

az ml job download --name <sweep-job> --all

Optionally, download only the best trial output:

az ml job download --name <sweep-job> --output-name model

References

Next steps

  • Last updated on