Inaccurate/Different Results in using Exported ONNX Object Detector vs Portal

Ben Knutson 6

Hi, my name is Ben Knutson, thanks so much for your time. I'm having a problem using an exported ONNX object detector in an ML.NET C# console application. Basically I've taken some sample code (provided by Microsoft) for a C# console application and I've modified the anchors and labels/classes, everything is running but the results I'm getting don't match what I'm seeing in the portal.. it's typically performing worse, but still working enough for me to know it's nothing obvious.

I think there's something happening pre/post-processing in Azure Custom Vision portal that has not ever been made public before. Either that or the exported ONNX models are just incorrect.. I'm wondering if someone can get in touch with me to resolve this issue?

romungi-MSFT 42,311 Reputation points Microsoft Employee

2021-01-04T07:03:37.957+00:00

@Ben Knutson I believe you are referring to this documentation mentioned here to export the model and use it directly with the application. I think in this case the exported model is not modified in the application. If you are using a different sample could you please post the link to the code or documentation?
Ben Knutson 6 Reputation points

2021-01-04T19:21:26.86+00:00

So the link you sent is for a UWP application, which may or may not be in working condition (I want to mention quickly that the source code there is for classification, a tutorial source code has yet to be created for object detection). But for my purposes I'm looking for a cross-platform webservice option so I'm attempting to start with a console C# .NET Core and work my way up to an ASP.NET Core. To do this I'm extending this example: https://github.com/dotnet/machinelearning-samples/tree/master/samples/csharp/end-to-end-apps/ObjectDetection-Onnx and changing a couple of the obvious things like the names of the input/output (if necessary) and the anchor points, and subbing in my ONNX model. It's semi working cause it will put bounding boxes around the objects, sometimes, but the confidence of the prediction is not the same, and sometimes it missed objects. I've seen others complaining about similar issues on forums, so I know it's not just me.
Ben Knutson 6 Reputation points

2021-01-04T19:22:51.877+00:00

Also want to mention - I tried a couple other export methods; I tried exporting as a tensorflow model and using python to make predictions, again I got it running but the predictions were different than the portal, though this time the predictions were sometimes higher accuracy, sometimes lower accuracy, but never exactly the same as the portal. I also tried exporting as a TFJS and tried running that, but that could not be run at all.

Overall it seems to be a common issue that exported models from Custom Vision don't give the same output as the portal, and this is a major problem. To answer whether I have source code to share, I haven't created a github for it but yes I do, and I can, but it's one of those things where you can see it yourself if you use the sample code provided (whether ONNX C# or Tensorflow Python).
romungi-MSFT 42,311 Reputation points Microsoft Employee

2021-01-07T11:51:55.543+00:00

@Ben Knutson You can post your code along with other details on github so others can advise on any improvements. If your data/model is classified and if you would not like to share it, you can always raise a support request from Azure portal against the custom vision resource. This will ensure you can share the details privately and an engineer is assigned to the issue who can help you 1:1 by working with our service team. If you do not have a valid support plan to raise an issue we can certainly help in this scenario by providing a one time free support ticket for this issue. Thanks!!
Ben Knutson 6 Reputation points

2021-01-10T05:37:28.023+00:00

Thanks so much! I don't have an Azure support plan, so a 1 time ticket would be amazing, thank you! Please let me know if you need me to do anything or provide contact information.

1 answer

Arif Budimartoyo 6 Reputation points

2022-08-15T00:18:01.61+00:00

@Ben Knutson You might need to double check the specifications on the exported model from Custom Vision at Azure Cognitive Service, and apply the same on the .NET ML. The standard ONNX model export from Azure Custom Vision is cropped with 224 x 224 size and converted to BGR as specified in the documentations below,

https://learn.microsoft.com/en-us/azure/machine-learning/how-to-inference-onnx-automl-image-models?tabs=multi-class#tabpanel_4_multi-class

https://learn.microsoft.com/en-us/samples/azure-samples/cognitive-services-onnx-customvision-sample/cognitive-services-onnx-customvision-sample/

I hope this will help.

Arif
Please sign in to rate this answer.

1 person found this answer helpful.

0 comments No comments
Sign in to comment