Early termination in AzureML CLI pipeline

Question

Early termination in AzureML CLI pipeline

matsuo_basho 30

I'm running a pipeline job using the AzureML cli syntax. https://learn.microsoft.com/en-us/azure/machine-learning/reference-yaml-job-pipeline?view=azureml-api-2 Is there a way to incorporate early stopping so that if the last epoch's loss (or whatever metric) doesn't improve above a certain threshold, then the job finishes?
I'm training on many epochs and would rather not waste compute resources (and waiting) if the metrics aren't improving.

dupammi 8,615 Reputation points Microsoft External Staff

2024-01-29T08:30:55.5866667+00:00
Hi @matsuo_basho , Thank you for the response. You are correct that the article I referenced is not directly related to the pipeline job. However, the EarlyStopping callback can be used in both SweepJob and pipeline jobs.

To incorporate early stopping in your pipeline job, you can use the MedianStoppingPolicy class from the azureml.train.hyperdrive module. This policy stops the job if the running job's performance falls outside the median of the running average of the primary metric. In your code, you can define the early_stopping_policy variable as follows:

early_stopping_policy = MedianStoppingPolicy(evaluation_interval=1, delay_evaluation=5)

This policy will evaluate the primary metric every evaluation_interval interval and stop the job if the running job's performance falls outside the median of the running average of the primary metric. The delay_evaluation parameter specifies the number of intervals to delay the first policy evaluation.You can then pass this policy to the HyperDriveConfig object as follows:

hyperdrive_config = HyperDriveConfig(run_config=script_config, hyperparameter_sampling=param_sampling, policy=early_stopping_policy, primary_metric_name='Accuracy', primary_metric_goal=PrimaryMetricGoal.MAXIMIZE, max_total_runs=20, max_concurrent_runs=4)

This will configure the hyperdrive run to use the early_stopping_policy policy to stop the job if the primary metric falls outside the median of the running average.

You can then submit your pipeline run as follows:

pipeline_run = experiment.submit(pipeline, tags={'Job type': 'Pipeline'})

This will submit your pipeline run to the Azure Machine Learning service. You can monitor the run's progress and view the results in the Azure Machine Learning portal using the link provided in the output.

I hope this helps. If this answers your query, do click Accept Answer and Yes for was this answer helpful in the below answer section.
matsuo_basho 30 Reputation points

2024-01-30T02:27:56.0266667+00:00

@dupammi thanks for the idea. I am not using the Python SDK, I am using the AzuremML CLI pipeline, as I indicated in my question. Does this early stopping policy option exist for CLI?
dupammi 8,615 Reputation points Microsoft External Staff

2024-01-30T06:08:03.8233333+00:00

Hi @matsuo_basho , Thanks for your reply. While the Azure CLI commands can be used to set up the overall pipeline, the implementation of the EarlyStopping callback can be handled within the training script. This allows for greater flexibility and customization in defining the early stopping policy based on the specific needs of the training process.

I hope this helps! Thank you for understanding.
matsuo_basho 30 Reputation points

2024-01-30T09:36:39.7133333+00:00

@dupammi I'm not using the Python SDK and so I think you mean to incorporate that early stopping feature into whatever API I'm using for training, in my case, it's the transformers syntax. In that case, this is no longer an Azure question. Am I understanding you correctly?
dupammi 8,615 Reputation points Microsoft External Staff

2024-01-30T10:02:34.7466667+00:00

Hi @matsuo_basho , Thank you for the response. Yes, you are correct. If you are not using the Python SDK and are instead using a different API for training (in your case, the transformers syntax), then the implementation of the EarlyStopping feature would depend on the specific API you are using. Hope this helps. Thank you for understanding.

1 answer

Your answer

matsuo_basho 30 Reputation points

2024-01-30T02:27:56.0266667+00:00

@dupammi thanks for the idea. I am not using the Python SDK, I am using the AzuremML CLI pipeline, as I indicated in my question. Does this early stopping policy option exist for CLI?
dupammi 8,615 Reputation points Microsoft External Staff

2024-01-30T06:08:03.8233333+00:00

Hi @matsuo_basho , Thanks for your reply. While the Azure CLI commands can be used to set up the overall pipeline, the implementation of the EarlyStopping callback can be handled within the training script. This allows for greater flexibility and customization in defining the early stopping policy based on the specific needs of the training process.

I hope this helps! Thank you for understanding.
matsuo_basho 30 Reputation points

2024-01-30T09:36:39.7133333+00:00

@dupammi I'm not using the Python SDK and so I think you mean to incorporate that early stopping feature into whatever API I'm using for training, in my case, it's the transformers syntax. In that case, this is no longer an Azure question. Am I understanding you correctly?
dupammi 8,615 Reputation points Microsoft External Staff

2024-01-30T10:02:34.7466667+00:00

Hi @matsuo_basho , Thank you for the response. Yes, you are correct. If you are not using the Python SDK and are instead using a different API for training (in your case, the transformers syntax), then the implementation of the EarlyStopping feature would depend on the specific API you are using. Hope this helps. Thank you for understanding.

Answer 1

dupammi 8,615 Microsoft External Staff

Hi @matsuo_basho ,

Thank you for using Microsoft Q&A forum.

It seems you can incorporate early stopping in your AzureML CLI pipeline job. AzureML provides several early termination policies that you can use to terminate a job if the primary metric doesn't improve above a certain threshold.

You can find more information about these policies in the AzureML documentation: azureml-api-1#early-termination and azureml-api-2#early-termination

Hope this helps.

If this answers your query, do click Accept Answer and Yes for was this answer helpful.

matsuo_basho 30 Reputation points

2024-01-25T19:09:06.0666667+00:00

@dupammi the article you reference is not a pipeline job, it is a SweepJob type, which isn't what I'm asking about.

Share via

Early termination in AzureML CLI pipeline

1 answer

Your answer