Form Recognizer - Issue with custom model table dynamic rows

Question

Form Recognizer - Issue with custom model table dynamic rows

Knox Lam 6

Hi,

We are currently delivering a solution for our client which involves using form recognizer custom model to scan invoices from different suppliers. The problem is sometimes the invoice may have certain tagged labels stained or they may not be present due to various reasons we cannot control.

We expect the returned results to be NULL for a tagged label if it does not exist on the invoice. However, it would instead return texts on the invoice we did not tag. While we could get around this for regular tags by tagging the labels we don't want, this method doesn't really work for tables with dynamic rows (see image below).

In the image below, when the first column (description) of the 2nd row in the table is empty, FR is unable to recognize this and returned a wrong table.
The layout didn't really change on the invoice, it's just that some elements are omitted and the invoice doesn't look distorted to the human eye.
We used 8 sample docs to train this model.

Is Form Recognizer designed to handle forms correctly when there are elements on the form omitted or some extra elements (like, maybe a small coffee stain or some ink) on the form?
Are there any tricks we can use in this scenario to improve the accuracy of the model?

Thanks!

0 comments

1 answer

Your answer

Answer 1

GiftA-MSFT 11,181 Moderator

Hi, here are some considerations:

As an experiment, try to train the custom model without labels to see if you get better results
Ensure your training data set follows the input requirements (pay attention to the additional tips as well)
Improve your model accuracy by labeling additional forms and retraining to create a new model (consider adding the prediction file to the training set, label it, and train again)
Try using the new Form Recognizer Studio (public preview) to see if you get better results

--- *Kindly Accept Answer if the information helps. Thanks.*

Knox Lam 6 Reputation points

2021-11-22T01:59:06.473+00:00

Hi,

Thanks for the response.
We are already following the input requirements for the data set and we have added more docs as well. Unfortunately, it didn't really help in this case.
The form recognizer studio is returning the exact same results as the labelling tool :\

At this point we just want to know whether Form Recognizer is designed to handle forms correctly when some tagged labels are missing (like a few labels, not all of them) from the form or there are some stains on the forms.

Thanks
GiftA-MSFT 11,181 Reputation points Moderator

2021-11-23T16:41:34.747+00:00

I confirmed the following from the product team. FR should be able to handle omitted elements or stains on the form. In this particular case, it looks like the table label is missing a column - it might be worth trying to include and label that column in the table label (even if the info for that column is not needed) and see if that helps improve the results.
Knox Lam 6 Reputation points

2021-11-24T02:54:14.183+00:00

Hi,

I added the supply date column between description and quantity.
That unfortunately didn't help. Both lines are still treated as 1 line and joined together.
Further tests seem to suggest we could omit the columns between the 1st column and the last column, may be its reliant on those columns?

Not sure if this is related but just for your info, the way we tagged the table is kinda funny as there can be 1-2 tables on the form with the exact same layout and the 2nd table is located right below the 1st table if it exists.
We tried to tag them with two separate table tags but the results were very inaccurate, but it worked fine when we tagged the line items from both tables on the invoice into 1 table tag.

Share via

Form Recognizer - Issue with custom model table dynamic rows

1 answer

Your answer