Custom NER detects all the numbers in a text as Invoice ID even though only 5* and A numbers were only used for training the invoice Ids

Question

Custom NER detects all the numbers in a text as Invoice ID even though only 5* and A numbers were only used for training the invoice Ids

Velliyangiri, Saravana Kumar 1

We have created a lis tof template texts for detecting invoice ids and we use the number series in question title to train a Custom NER model but still it detects numbers of any length as invoice ID as well combinations of number text and symbol as invoice ids. Any suggestions would be highly appreciated which would help here to improve the model so that numbers only in specific range or pattern provided in the training gets extracted as valid invoice ids

Ramr-msft 17,826 Reputation points

2022-12-30T03:07:04.063+00:00

@Velliyangiri, Saravana Kumar Thanks for the question. Use regular expressions to specify the pattern that the invoice IDs should follow. This can help the model focus on extracting only invoice IDs that match the specified pattern, rather than extracting any number that appears in the text.
Velliyangiri, Saravana Kumar 1 Reputation point

2022-12-30T11:52:38.94+00:00

@Ramr-msft Can you please help us with more details on how the regular expression can be used in training data while training the model? Any document or blog which would explain the steps here would be really helpful
Ramr-msft 17,826 Reputation points

2023-01-02T12:03:00.187+00:00
Sure! Regular expressions (regex) are a powerful tool for matching patterns in text. You can use them to specify the pattern of the invoice IDs you want the model to recognize in your training data.

Here's an example of how you could use a regular expression to match invoice IDs that consist of 5 digits followed by a letter:
import re

pattern = r"\d{5}[A-Z]" re.match(pattern, "12345A") # returns a match object re.match(pattern, "12345") # returns None re.match(pattern, "12345a") # returns None

To use regular expressions in your training data, you will need to annotate the text with the appropriate entity labels. For example, if you have the following text:

"The invoice ID is 12345A and the total amount is $100."

You could annotate it like this:

The invoice ID is [12345A](InvoiceID) and the total amount is $100.

Then, you can use the regular expression pattern \d{5}[A-Z] to extract the invoice ID from the text.

Here are a few resources that you might find helpful for learning more about using regex in NER models:

This tutorial from spaCy's documentation explains how to use regex patterns to match specific types of entities: https://spacy.io/usage/rule-based-matching#regex
This tutorial from the Stanford NLP Group explains how to use regex patterns in a NER model built with the Stanford NER tool: https://nlp.stanford.edu/software/regexner.html
Velliyangiri, Saravana Kumar 1 Reputation point

2023-01-03T15:36:15.637+00:00

@Ramr-msft Thanks for the suggestion and detailed explanation together with the documentation. We have tried using regex pattern in training data for training custom named entity recognition model from Azure cognitive services. A sample input text is as shown below.

Monthly charges in my invoice 5000000138 is wrong

We have provided 200 files as part of training data with similar numerical only invoice ids in same pattern starting with 5 and in project created through rest API we have provided entity offset details as well but still model was extracting invoice ids of lesser length like 3 digits, alphanumeric ids as well as it extracted only partial numbers if the id is a combination of alphabets, special characters and numbers as in case of value 2$3#1.

Can you please suggest if in case we are missing anything here or any other improvement in training data can be done to improve the model performance?

Thanks.
Velliyangiri, Saravana Kumar 1 Reputation point

2023-01-11T15:31:04.87+00:00

Any suggestions here would be highly appreciated as we are still facing the same model behavior even after following the regex approach
Ramr-msft 17,826 Reputation points

2023-01-12T04:41:45.0133333+00:00

@Velliyangiri, Saravana Kumar Can you please share the pattern that you have tried?

Your answer

Ramr-msft 17,826 Reputation points

2022-12-30T03:07:04.063+00:00

@Velliyangiri, Saravana Kumar Thanks for the question. Use regular expressions to specify the pattern that the invoice IDs should follow. This can help the model focus on extracting only invoice IDs that match the specified pattern, rather than extracting any number that appears in the text.
Velliyangiri, Saravana Kumar 1 Reputation point

2022-12-30T11:52:38.94+00:00

@Ramr-msft Can you please help us with more details on how the regular expression can be used in training data while training the model? Any document or blog which would explain the steps here would be really helpful
Ramr-msft 17,826 Reputation points

2023-01-02T12:03:00.187+00:00

Sure! Regular expressions (regex) are a powerful tool for matching patterns in text. You can use them to specify the pattern of the invoice IDs you want the model to recognize in your training data.

Here's an example of how you could use a regular expression to match invoice IDs that consist of 5 digits followed by a letter:
import re

pattern = r"\d{5}[A-Z]" re.match(pattern, "12345A") # returns a match object re.match(pattern, "12345") # returns None re.match(pattern, "12345a") # returns None

To use regular expressions in your training data, you will need to annotate the text with the appropriate entity labels. For example, if you have the following text:

"The invoice ID is 12345A and the total amount is $100."

You could annotate it like this:

The invoice ID is [12345A](InvoiceID) and the total amount is $100.

Then, you can use the regular expression pattern \d{5}[A-Z] to extract the invoice ID from the text.

Here are a few resources that you might find helpful for learning more about using regex in NER models:

This tutorial from spaCy's documentation explains how to use regex patterns to match specific types of entities: https://spacy.io/usage/rule-based-matching#regex
This tutorial from the Stanford NLP Group explains how to use regex patterns in a NER model built with the Stanford NER tool: https://nlp.stanford.edu/software/regexner.html
Velliyangiri, Saravana Kumar 1 Reputation point

2023-01-03T15:36:15.637+00:00

@Ramr-msft Thanks for the suggestion and detailed explanation together with the documentation. We have tried using regex pattern in training data for training custom named entity recognition model from Azure cognitive services. A sample input text is as shown below.

Monthly charges in my invoice 5000000138 is wrong

We have provided 200 files as part of training data with similar numerical only invoice ids in same pattern starting with 5 and in project created through rest API we have provided entity offset details as well but still model was extracting invoice ids of lesser length like 3 digits, alphanumeric ids as well as it extracted only partial numbers if the id is a combination of alphabets, special characters and numbers as in case of value 2$3#1.

Can you please suggest if in case we are missing anything here or any other improvement in training data can be done to improve the model performance?

Thanks.
Velliyangiri, Saravana Kumar 1 Reputation point

2023-01-11T15:31:04.87+00:00

Any suggestions here would be highly appreciated as we are still facing the same model behavior even after following the regex approach
Ramr-msft 17,826 Reputation points

2023-01-12T04:41:45.0133333+00:00

@Velliyangiri, Saravana Kumar Can you please share the pattern that you have tried?

Share via

Custom NER detects all the numbers in a text as Invoice ID even though only 5********* and ****A******** numbers were only used for training the invoice Ids