Metrics for Azure Custom Vision and default value for Overlap Threshold when using Prediction API

Question

Metrics for Azure Custom Vision and default value for Overlap Threshold when using Prediction API

Tom Jenkins 46

Hello,
I'm using Azure Custom Vision for a research project.

I would like to know some things about the Model:

How does the model calculate the average precision?
When using the the prediction API, what are the settings for the Overlap Threshold?

(If i set the Overlap Threshold to 100 an send an Image via Prediction API to the Endpoint, i still get results, although the Precision and Recall are 0. That's why i think there must be a default value for the Overlap Threshold, that cannot be changed. Does that make sense? If yes, my question would be:
What is the default value for the Overlap Threshold, when posting via Predicition API?)

Thank you for the Support.

Ramr-msft 17,826 Reputation points

2021-01-05T16:52:54.19+00:00

@Tom Jenkins Thanks for the question. We are investigating and will update with the details.

Accepted answer

0 additional answers

Your answer

Ramr-msft 17,826 Reputation points

2021-01-05T16:52:54.19+00:00

@Tom Jenkins Thanks for the question. We are investigating and will update with the details.

Answer 1

GiftA-MSFT 11,176

Hi, mean average precision is the average value of the average precision (AP). AP is the area under the precision/recall curve (precision plotted against recall for each prediction made). It seems you are referring to the Overlap Threshold for Get Iteration Performance API. If so, the default value if you didn't provide any is 0.5. However, from our portal we set the threshold to 0.3 when making a call. Hope this helps.

Tom Jenkins 46 Reputation points

2021-01-06T10:47:39.353+00:00

Hello @GiftA-MSFT ,
Thank you for the answer.

It would be nice, if there's a publication where the average precision is defined, although it might be not that easy to find out.

Also, I would appreciate if the Overlap Threshold of 0.3 would be mentioned in the Documentation (if it is not anywhere i did not look).

Thank you for the Support.
Tom Jenkins 46 Reputation points

2021-01-06T11:39:32.263+00:00

I found this recently published paper:
https://ieeexplore.ieee.org/abstract/document/9145130
(can also be found here: https://github.com/rafaelpadilla/Object-Detection-Metrics)
Which one of the definitions of Average Precision (AP) might be the most fitting?
GiftA-MSFT 11,176 Reputation points

2021-01-06T20:11:08.92+00:00

Please review my updated response above. I have also shared your feedback with our content team to enhance the document with more information about the metrics. Thanks.

Share via

Metrics for Azure Custom Vision and default value for Overlap Threshold when using Prediction API

0 additional answers

Your answer