How to improve the accuracy and recall of exported ONNX model?

Mark Oueis 1

We have a custom vision model that we are extremely happy with - built from the ground up using custom vision. Because we need to do detections every second, we decided to go on the edge and export the model to ONNX (otherwise on the cloud the cost is prohibitive). We switched our model to a "compact" domain and then exported it to ONNX and ran it from a c# console application on windows 10. We expected worse accuracy, but the results were quite terrible - significantly worse that they were when using the [a1] domain. It would miss many objects or find them with very low probability.

We then trained the compact model a little more to see if we can improve the recall. It seemed to help a little, but still nowhere near what it was. When we use the API against the compact domain (as opposed to export and running it from the c# model) we get better results (though still not as good). The worst is when we just run it from our c# app.

How can I improve the performance of this exported to ONNX model running from our c# app

Are we doing something wrong on the inference side from the console app. What should we look for?
Will more training help?
Can we export a larger but more accurate model
Could it be that this is as good as it gets because its an exported compact domain?
Are there other options we haven't considered

I'm thinking if we should give us on custom vision export and train our own model with a different platform or if we can continue on custom vision and get the detection accuracy we need.

Thank you

romungi-MSFT 46,651 Reputation points Microsoft Employee

2022-08-25T10:01:38.087+00:00
@Mark Oueis Based on previous experience, I could suggest the following recommendations for the above questions.

Are we doing something wrong on the inference side from the console app. What should we look for?

The usage of the exported model is similar to what is mentioned in this sample. The sample does mention the response from the exported model will be different than the cloud API due to the image pre-processing logic with C#

Will more training help?

Yes, more training will help if you wish to export the model and use it instead of the cloud API.

Can we export a larger but more accurate model

Currently, only compact domains can be used for export. The general model is not available for export.

Could it be that this is as good as it gets because its an exported compact domain?

Yes, but you could add more training data in your project and train and export with compact domain.

Are there other options we haven't considered

You could try to use the model with python and check if better results are seen since the image pre-processing is different for both.

I hope this helps!!
Mark Oueis 1 Reputation point

2022-08-25T13:49:43.103+00:00

Hi Romungi, thank you very much for your quick response!

I read that the exported model will be different, but its so different that I think maybe we're doing something wrong. There are a few samples online and maybe we just used a wrong one. The one we used looks like this - https://github.com/jwood803/MLNetExamples/blob/master/MLNetExamples/CustomVisionOnnx/Program.cs which appears slightly different than what you suggested above. Is that the same thing or is your suggestion better?

Also it appears that sample we want has a broken link? See image below

Is that the link we want? What would be your recommendation for the best code to follow to get the most accurate results. We are using compact (not compact s1) and we are doing object detection.

Thank you

Share via

How to improve the accuracy and recall of exported ONNX model?

Your answer