Azure Custom Classification Model - accuracy, precision, recall and f1

Question

Azure Custom Classification Model - accuracy, precision, recall and f1

Roberto Araujo Filho 115

Using Azure, when training a Custom Classification Model, is there a way to set a group of documents as a test set and automatic get accuracy, precision, recall and f1 score metrics to evaluate the model´s performance?

navba-MSFT 27,550 Reputation points Microsoft Employee Moderator

2024-03-06T03:13:02.7433333+00:00
@Roberto Araujo Filho Welcome to Microsoft Q&A Forum, Thank you for posting your query here!

When training a Custom Classification Model in Azure AI Document Intelligence, you can indeed set a group of documents as a test set and automatically get accuracy, precision, recall, and F1 score metrics to evaluate the model’s performance.

Here’s how it works:

Your dataset is split into two parts: a set for training, and a set for testing.

The training set is used to train the model, while the testing set is used as a test for the model after training to calculate the model performance and evaluation.

The testing set isn’t introduced to the model through the training process, to make sure that the model is tested on new data.

Model evaluation is triggered automatically after training is completed successfully.

The evaluation process starts by using the trained model to predict user-defined classes for documents in the test set, and compares them with the provided data tags (which establishes a baseline of truth).

The results are returned so you can review the model’s performance.

For evaluation, custom text classification uses the following metrics:

Precision: Measures how precise/accurate your model is. It’s the ratio between the correctly identified positives (true positives) and all identified positives. The precision metric reveals how many of the predicted classes are correctly labeled. Precision = #True_Positive / (#True_Positive + #False_Positive)

Recall: Measures the model’s ability to predict actual positive classes. It’s the ratio between the predicted true positives and what was actually tagged. The recall metric reveals how many of the predicted classes are correct. Recall = #True_Positive / (#True_Positive + #False_Negatives)

F1 score: The F1 score is a function of Precision and Recall. It’s needed when you seek a balance between Precision and Recall. F1 Score = 2 * Precision * Recall / (Precision + Recall)

Note that Precision, recall, and F1 score are calculated for each class separately (class-level evaluation) and for the model collectively (model-level evaluation).

More info here and here. Hope this helps.
Roberto Araujo Filho 115 Reputation points

2024-03-06T10:45:43.8933333+00:00
Hello @navba-MSFT ,

Thanks for the prompt reply. You sent 2 links in your answer:

One of them shows how to build and train a custom model using "Azure Document Intelligence", but does not show how to automatic get the evaluation metrics;

The other link, shows how to automatic get the evaluation metrics... but for a model created inside "Azure Language Studio"...

I would like to know if there is an automatic metric´s calculation for a Custom Classification Model created inside "Azure Document Intelligence", just the same way there is in "Azure Language Studio". Or if using "Azure Document Intelligence" I have to do the tests and calculate metrics by myself...
navba-MSFT 27,550 Reputation points Microsoft Employee Moderator

2024-03-11T04:47:50.82+00:00

@Roberto Araujo Filho Thanks for getting back. There is an automatic metric calculation for a Custom Classification Model created inside Azure Document Intelligence. After the model is trained successfully, the evaluation metrics are calculated automatically.

The evaluation process starts by using the trained model to predict user-defined classes for documents in the test set, and compares them with the provided data tags (which establishes a baseline of truth). The results are returned so you can review the model’s performance.

The evaluation metrics include precision, recall, and F1 score, which are calculated for each class separately (class-level evaluation) and for the model collectively (model-level evaluation).

You can refer to the following link for more information on evaluation metrics in Azure Document Intelligence:

https://docs.microsoft.com/en-us/azure/ai-services/document-intelligence/how-to-guides/build-a-custom-model?view=doc-intel-4.0.0#evaluate-the-model-performance
Roberto Araujo Filho 115 Reputation points

2024-03-11T10:26:55.3166667+00:00

Hello @navba-MSFT ,

Thank you again for you response. Unfortunately I couldn´t find the metric calculation you mentioned in Azure Document Intelligence. It doesn't even give the option to divide the document samples into train and test sets. As far as I could observe there is no such feature, so I calculated the metrics using code.

Your answer

Roberto Araujo Filho 115 Reputation points

2024-03-06T10:45:43.8933333+00:00

Hello @navba-MSFT ,

Thanks for the prompt reply. You sent 2 links in your answer:

One of them shows how to build and train a custom model using "Azure Document Intelligence", but does not show how to automatic get the evaluation metrics;

The other link, shows how to automatic get the evaluation metrics... but for a model created inside "Azure Language Studio"...

I would like to know if there is an automatic metric´s calculation for a Custom Classification Model created inside "Azure Document Intelligence", just the same way there is in "Azure Language Studio". Or if using "Azure Document Intelligence" I have to do the tests and calculate metrics by myself...
navba-MSFT 27,550 Reputation points Microsoft Employee Moderator

2024-03-11T04:47:50.82+00:00

@Roberto Araujo Filho Thanks for getting back. There is an automatic metric calculation for a Custom Classification Model created inside Azure Document Intelligence. After the model is trained successfully, the evaluation metrics are calculated automatically.

The evaluation process starts by using the trained model to predict user-defined classes for documents in the test set, and compares them with the provided data tags (which establishes a baseline of truth). The results are returned so you can review the model’s performance.

The evaluation metrics include precision, recall, and F1 score, which are calculated for each class separately (class-level evaluation) and for the model collectively (model-level evaluation).

You can refer to the following link for more information on evaluation metrics in Azure Document Intelligence:

https://docs.microsoft.com/en-us/azure/ai-services/document-intelligence/how-to-guides/build-a-custom-model?view=doc-intel-4.0.0#evaluate-the-model-performance
Roberto Araujo Filho 115 Reputation points

2024-03-11T10:26:55.3166667+00:00

Hello @navba-MSFT ,

Thank you again for you response. Unfortunately I couldn´t find the metric calculation you mentioned in Azure Document Intelligence. It doesn't even give the option to divide the document samples into train and test sets. As far as I could observe there is no such feature, so I calculated the metrics using code.

Share via

Azure Custom Classification Model - accuracy, precision, recall and f1

Your answer