How to train custom model for different document formats with the same set of labels?

Hugo Scaramal 41

I'm trying to understand what is the best way to train a custom model for invoices in languages not supported by the prebuilt invoice model, french as an example.

As normal we will have many different invoice formats from different vendors, but in all of them, we will extract the same set of labels (invoice number, amount, date, vendor name, etc).

Should I create a model per vendor and compose it?
If I do so, do I need to train it for all vendors, or will it work for invoices that were not trained, but use the same verbiage as trained invoices?

Ramr-msft 17,616 Reputation points

2021-04-26T07:28:03.437+00:00

@Hugo Scaramal Thanks for the question. We are checking internally with the product team for best practices for many different invoice formats from different vendors.
When incoming forms belongs to one of several templates, Model Compose allows multiple models to be composed and called with a single model ID. When a document is submitted to be analyzed with a composed model ID, a classification step is first performed to route it to the correct custom model.

Knowledge Extraction Recipe:https://github.com/microsoft/knowledge-extraction-recipes-forms
Hugo Scaramal 41 Reputation points

2021-04-27T13:30:55.657+00:00

Hi, thank you for your reply.

I was hoping that by training a single model, with enough variation, that it would be possible to analyze invoices that weren't trained as well.
Much like what I assume the built-in invoice model does. I could be wrong, but I wouldn't think that Microsoft trained every single invoice layout to have the built-in invoice model to work. And most likely it was trained based on verbiage around the information it needs to recognize using machine learning.
So my thought process on the custom model training was around that same idea.

But, the more I learn, the more it indicates I'm might be on the wrong path, and I will have to train every single layout that I want the custom model to analyze.
So, I'm looking forward to some guidance on this specific scenario (invoices in languages other than English).

And thanks for sharing the Knowledge Extraction Recipe, there is a lot of interesting information on that document.

Accepted answer

Ramr-msft 17,616 Reputation points

2021-05-04T06:56:19.7+00:00

@Hugo Scaramal Thanks, For invoices you should use the pre-built Invoice model, no training required - https://learn.microsoft.com/en-us/azure/cognitive-services/form-recognizer/concept-invoices.If you need to train a model and not use the pre-built than yes a model per vendor \ provider and compose them. Start with the top providers so that you get more coverage.
Please sign in to rate this answer.
Hugo Scaramal 41 Reputation points

2021-05-04T11:59:24.543+00:00

Thank you.
We will use the pre-built for English/US invoices, and I'll train for non-English invoices as recommended.
I ended up writing a little console app to help me coordinate model training/composing since we will have multiple models and composed models.
Basically one compose model per country/language.
Thanks again, I appreciate your help.
Sign in to comment

How to train custom model for different document formats with the same set of labels?

0 additional answers