Tokenizer Constructors
Definition
Important
Some information relates to prerelease product that may be substantially modified before it’s released. Microsoft makes no warranties, express or implied, with respect to the information provided here.
Overloads
| Tokenizer() |
Initializes a new instance of the Tokenizer class. |
| Tokenizer(Model, PreTokenizer, Normalizer) |
Create a new Tokenizer object. |
Tokenizer()
- Source:
- Tokenizer.cs
- Source:
- Tokenizer.cs
- Source:
- Tokenizer.cs
Initializes a new instance of the Tokenizer class.
protected Tokenizer();Protected Sub New ()Applies to
Tokenizer(Model, PreTokenizer, Normalizer)
Create a new Tokenizer object.
public Tokenizer(Microsoft.ML.Tokenizers.Model model, Microsoft.ML.Tokenizers.PreTokenizer? preTokenizer = default, Microsoft.ML.Tokenizers.Normalizer? normalizer = default);new Microsoft.ML.Tokenizers.Tokenizer : Microsoft.ML.Tokenizers.Model * Microsoft.ML.Tokenizers.PreTokenizer * Microsoft.ML.Tokenizers.Normalizer -> Microsoft.ML.Tokenizers.TokenizerPublic Sub New (model As Model, Optional preTokenizer As PreTokenizer = Nothing, Optional normalizer As Normalizer = Nothing)Parameters
- model
- Model
The Model in use by the Tokenizer.
- preTokenizer
- PreTokenizer
The optional PreTokenizer in use by the Tokenizer. WhiteSpace PreTokenizer will be used if this parameter is null.
- normalizer
- Normalizer
The optional Normalizer in use by the Tokenizer.