Azure Automated ML(interface) choosing primary metrics to handle imbalanced data

J. Jeong 61 Reputation points
2020-06-29T02:34:21.863+00:00

I figured out that there are some primary metrics I can choose when I run an automated ML experiment. Yet the number of primary metrics is fewer than the run metrics in the result page. I want to deal with imbalanced data(10:1 or 20:1) and

looked up the links below:
https://learn.microsoft.com/en-us/azure/machine-learning/concept-manage-ml-pitfalls#identify-models-with-imbalanced-data
and
https://learn.microsoft.com/en-us/azure/machine-learning/how-to-configure-auto-train

It seems F1 score is recommended to evaluate each model with imbalanced data.

Here are my questions:

  • Is there any way to set F1 score or multiple measures as a primary metric?
  • If there is no such way, should I do it manually?
  • Of all the given primary metrics, which primary metric is the most appropriate(to build a Classification model with imbalanced data)?

Thanks.

Azure Machine Learning
Azure Machine Learning
An Azure machine learning service for building and deploying models.
2,718 questions
{count} vote

Accepted answer
  1. Ramr-msft 17,731 Reputation points
    2020-06-30T09:20:12.347+00:00

    For imbalanced data, it is preferred to choose AUC Weighted. Also user should then choose a metric that is appropriate to work well for imbalance. E.g. F1, micro averaged AUC, balanced accuracy for model evaluation. For primary metric (metric used for model optimization) the user should preferably choose AUC Weighted instead of accuracy.
    Currently from the ml.azure.com the following metrics are supported. To add F1 score metric forwarded to product team to check on this.
    https://learn.microsoft.com/en-us/azure/machine-learning/how-to-configure-auto-train#primary-metric
    10986-screenshot-162.png


0 additional answers

Sort by: Most helpful