BanditPolicy Class
Defines an early termination policy based on slack criteria, and a frequency and delay interval for evaluation.
Initialize a BanditPolicy with slack factor, slack_amount, and evaluation interval.
- Inheritance
-
azureml.train.hyperdrive.policy.EarlyTerminationPolicyBanditPolicy
Constructor
BanditPolicy(evaluation_interval=1, slack_factor=None, slack_amount=None, delay_evaluation=0)
Parameters
- slack_factor
- float
The ratio used to calculate the allowed distance from the best performing experiment run.
- delay_evaluation
- int
The number of intervals for which to delay the first policy evaluation.
If specified, the policy applies every multiple of evaluation_interval
that is greater than or equal to delay_evaluation
.
- slack_factor
- float
The ratio used to calculate the allowed distance from the best performing experiment run.
- delay_evaluation
- int
The number of intervals for which to delay the first policy evaluation.
If specified, the policy applies every multiple of evaluation_interval
that is greater than or equal to delay_evaluation
.
Remarks
The Bandit policy takes the following configuration parameters:
slack_factor
: The amount of slack allowed with respect to the best performing training run. This factor specifies the slack as a ratio.slack_amount
: The amount of slack allowed with respect to the best performing training run. This factor specifies the slack as an absolute amount.evaluation_interval
: Optional. The frequency for applying the policy. Each time the training script logs the primary metric counts as one interval.delay_evaluation
: Optional. The number of intervals to delay policy evaluation. Use this parameter to avoid premature termination of training runs. If specified, the policy applies every multiple ofevaluation_interval
that is greater than or equal todelay_evaluation
.
Any run that doesn't fall within the slack factor or slack amount of the evaluation metric with respect to the best performing run will be terminated.
Consider a Bandit policy with slack_factor
= 0.2 and evaluation_interval
= 100.
Assume that run X is the currently best performing run with an AUC (performance metric) of 0.8 after 100
intervals. Further, assume the best AUC reported for a run is Y. This policy compares the value
(Y + Y * 0.2) to 0.8, and if smaller, cancels the run. If delay_evaluation
= 200, then the
first time the policy will be applied is at interval 200.
Now, consider a Bandit policy with slack_amount
= 0.2 and evaluation_interval
= 100.
If Run 3 is the currently best performing run with an AUC (performance metric) of 0.8 after 100 intervals,
then any run with an AUC less than 0.6 (0.8 - 0.2) after 100 iterations will be terminated.
Similarly, the delay_evaluation
can also be used to delay the first termination policy
evaluation for a specific number of sequences.
For more information about applying early termination policies, see Tune hyperparameters for your model.
Attributes
delay_evaluation
Return the number of sequences for which the first evaluation is delayed.
Returns
The delay evaluation.
Return type
evaluation_interval
slack_factor
Return the slack factor with respect to the best performing training run.
Returns
The slack factor.
Return type
POLICY_NAME
POLICY_NAME = 'Bandit'
Feedback
https://aka.ms/ContentUserFeedback.
Coming soon: Throughout 2024 we will be phasing out GitHub Issues as the feedback mechanism for content and replacing it with a new feedback system. For more information see:Submit and view feedback for