AI Search Microsoft analyzer en.microsoft token 'NN240' what does nn mean?

Question

AI Search Microsoft analyzer en.microsoft token 'NN240' what does nn mean?

chris.c.zhao chris.c.zhao 25

AI Search Microsoft analyzer en.microsoft token 'nn240' what does nn mean? the input is '240-47-7007', output: [{'token': '240-47-7007', 'startOffset': 0, 'endOffset': 11, 'position': 0}, {'token': '240', 'startOffset': 0, 'endOffset': 3, 'position': 0}, {'token': 'nn240', 'startOffset': 0, 'endOffset': 3, 'position': 0}, {'token': '47', 'startOffset': 4, 'endOffset': 6, 'position': 1}, {'token': 'nn47', 'startOffset': 4, 'endOffset': 6, 'position': 1}, {'token': '7007', 'startOffset': 7, 'endOffset': 11, 'position': 2}, {'token': 'nn7007', 'startOffset': 7, 'endOffset': 11, 'position': 2}] what does nn means inn 'token': 'nn47' ?

VenkateshDodda-MSFT 25,111 Reputation points Microsoft Employee Moderator

2024-02-07T11:04:15.12+00:00
@chris.c.zhao chris.c.zhao Thanks for reaching out to Microsoft Q&A, apologize for any inconvenience caused on this.

Could you please share more details whether you are using custom analyzers or language analyzer to understand better ?

Also, help us what exactly you are trying to do and is there any documentation steps you are following if yes could please help share us to check and assist you further.
chris.c.zhao chris.c.zhao 25 Reputation points

2024-02-07T14:46:00.6366667+00:00

my question is what is the meaning of nn in 'token': 'nn7007' when using Microsoft analyzer en.microsoft
VenkateshDodda-MSFT 25,111 Reputation points Microsoft Employee Moderator

2024-02-09T03:55:26.52+00:00

Hello @chris.c.zhao chris.c.zhao ,
Following up to see if you have a chance to check my previous response and helped. Do let us know if you have any further questions on this.

1 answer

Your answer

VenkateshDodda-MSFT 25,111 Reputation points Microsoft Employee Moderator

2024-02-07T11:04:15.12+00:00

@chris.c.zhao chris.c.zhao Thanks for reaching out to Microsoft Q&A, apologize for any inconvenience caused on this.

Could you please share more details whether you are using custom analyzers or language analyzer to understand better ?

Also, help us what exactly you are trying to do and is there any documentation steps you are following if yes could please help share us to check and assist you further.
chris.c.zhao chris.c.zhao 25 Reputation points

2024-02-07T14:46:00.6366667+00:00

my question is what is the meaning of nn in 'token': 'nn7007' when using Microsoft analyzer en.microsoft
VenkateshDodda-MSFT 25,111 Reputation points Microsoft Employee Moderator

2024-02-09T03:55:26.52+00:00

Hello @chris.c.zhao chris.c.zhao ,
Following up to see if you have a chance to check my previous response and helped. Do let us know if you have any further questions on this.

Answer 1

VenkateshDodda-MSFT 25,111 Microsoft Employee Moderator

@chris.c.zhao chris.c.zhao Thanks for your patience on this. I have checked with internal team and they has confirmed that nn means normalizednumber.

A tangible example of this would be numbers with formatting characters. Both the original value and "nn" version are stored as tokens. For example, 1,200 becomes the following tokens:

"tokens": [
        {
            "token": "1,200",
            "startOffset": 0,
            "endOffset": 5,
            "position": 0
        },
        {
            "token": "nn1200",
            "startOffset": 0,
            "endOffset": 5,
            "position": 0
        }
    ]

chris.c.zhao chris.c.zhao 25 Reputation points

2024-02-09T21:33:58.42+00:00

thanks a lot, you can close it.
VenkateshDodda-MSFT 25,111 Reputation points Microsoft Employee Moderator

2024-02-10T01:12:17.9366667+00:00

chris.c.zhao chris.c.zhao Thanks for your response. I have converted my comment to answer.
Please accept as Yes if the answer provided is useful, so that you can help others in the community looking for remediation for similar issues.

Share via

AI Search Microsoft analyzer en.microsoft token 'NN240' what does nn mean?

1 answer

Your answer