Share via


ClassicTokenizer Class

public final class ClassicTokenizer
extends LexicalTokenizer

Grammar-based tokenizer that is suitable for processing most European-language documents. This tokenizer is implemented using Apache Lucene.

Constructor Summary

Constructor Description
ClassicTokenizer(String name)

Creates an instance of ClassicTokenizer class.

Method Summary

Modifier and Type Method and Description
static ClassicTokenizer fromJson(JsonReader jsonReader)

Reads an instance of ClassicTokenizer from the JsonReader.

Integer getMaxTokenLength()

Get the maxTokenLength property: The maximum token length.

String getOdataType()

Get the odataType property: A URI fragment specifying the type of tokenizer.

ClassicTokenizer setMaxTokenLength(Integer maxTokenLength)

Set the maxTokenLength property: The maximum token length.

JsonWriter toJson(JsonWriter jsonWriter)

Methods inherited from LexicalTokenizer

Methods inherited from java.lang.Object

Constructor Details

ClassicTokenizer

public ClassicTokenizer(String name)

Creates an instance of ClassicTokenizer class.

Parameters:

name - the name value to set.

Method Details

fromJson

public static ClassicTokenizer fromJson(JsonReader jsonReader)

Reads an instance of ClassicTokenizer from the JsonReader.

Parameters:

jsonReader - The JsonReader being read.

Returns:

An instance of ClassicTokenizer if the JsonReader was pointing to an instance of it, or null if it was pointing to JSON null.

Throws:

IOException

- If the deserialized JSON object was missing any required properties.

getMaxTokenLength

public Integer getMaxTokenLength()

Get the maxTokenLength property: The maximum token length. Default is 255. Tokens longer than the maximum length are split. The maximum token length that can be used is 300 characters.

Returns:

the maxTokenLength value.

getOdataType

public String getOdataType()

Get the odataType property: A URI fragment specifying the type of tokenizer.

Overrides:

ClassicTokenizer.getOdataType()

Returns:

the odataType value.

setMaxTokenLength

public ClassicTokenizer setMaxTokenLength(Integer maxTokenLength)

Set the maxTokenLength property: The maximum token length. Default is 255. Tokens longer than the maximum length are split. The maximum token length that can be used is 300 characters.

Parameters:

maxTokenLength - the maxTokenLength value to set.

Returns:

the ClassicTokenizer object itself.

toJson

public JsonWriter toJson(JsonWriter jsonWriter)

Overrides:

ClassicTokenizer.toJson(JsonWriter jsonWriter)

Parameters:

jsonWriter

Throws:

Applies to