Token Class

Definition

Namespace:

Microsoft.ML.Tokenizers

Assembly:

Microsoft.ML.Tokenizers.dll

Package:

Microsoft.ML.Tokenizers v0.21.1

Represent the token produced from the tokenization process containing the token substring, the id associated to the token substring, and the offset mapping to the original string.

public sealed class Token

type Token = class

Public NotInheritable Class Token

Inheritance

Object

Token

Constructors

Token(Int32, String, ValueTuple<Int32,Int32>)

Construct a new Token object using the token value, Id, and the offset mapping to the original string.

Properties

Id	Gets or sets the Id value associated to the token.
Offset	Gets or sets the offset mapping to the original string.
Value	Gets or sets the token string value.