Share via

MicrosoftLanguageTokenizer interface

  • Reference
Package:
@azure/search-documents

Divides text using language-specific rules.

Extends
BaseLexicalTokenizer

Properties

isSearchTokenizer

A value indicating how the tokenizer is used. Set to true if used as the search tokenizer, set to false if used as the indexing tokenizer. Default is false.
language

The language to use. The default is English.
maxTokenLength

The maximum token length. Tokens longer than the maximum length are split. Maximum token length that can be used is 300 characters. Tokens longer than 300 characters are first split into tokens of length 300 and then each of those tokens is split based on the max token length set. Default is 255.
odatatype

Polymorphic discriminator, which specifies the different types this object can be

Inherited Properties

name

The name of the tokenizer. It must only contain letters, digits, spaces, dashes or underscores, can only start and end with alphanumeric characters, and is limited to 128 characters.

Property Details

isSearchTokenizer

A value indicating how the tokenizer is used. Set to true if used as the search tokenizer, set to false if used as the indexing tokenizer. Default is false.

isSearchTokenizer?: boolean

Property Value

boolean

language

The language to use. The default is English.

language?: MicrosoftTokenizerLanguage

Property Value

MicrosoftTokenizerLanguage

maxTokenLength

The maximum token length. Tokens longer than the maximum length are split. Maximum token length that can be used is 300 characters. Tokens longer than 300 characters are first split into tokens of length 300 and then each of those tokens is split based on the max token length set. Default is 255.

maxTokenLength?: number

Property Value

number

odatatype

Polymorphic discriminator, which specifies the different types this object can be

odatatype: "#Microsoft.Azure.Search.MicrosoftLanguageTokenizer"

Property Value

"#Microsoft.Azure.Search.MicrosoftLanguageTokenizer"

Inherited Property Details

name

The name of the tokenizer. It must only contain letters, digits, spaces, dashes or underscores, can only start and end with alphanumeric characters, and is limited to 128 characters.

name: string

Property Value

string

Inherited From BaseLexicalTokenizer.name