AddedToken Struct

Reference

Definition

Namespace:: Microsoft.ML.Tokenizers

Assembly:: Microsoft.ML.Tokenizers.dll

Package:: Microsoft.ML.Tokenizers v0.21.1

Important

Some information relates to prerelease product that may be substantially modified before it’s released. Microsoft makes no warranties, express or implied, with respect to the information provided here.

Represent a token added by the user on top of the existing Model vocabulary. AddedToken can be configured to specify the behavior they should have in various situations like:

Whether they should only match single words
Whether to include any WhiteSpace on its left or right

public struct AddedToken : IEquatable<Microsoft.ML.Tokenizers.AddedToken>

type AddedToken = struct

Public Structure AddedToken
Implements IEquatable(Of AddedToken)

Inheritance: ValueType
AddedToken

Implements: IEquatable<AddedToken>

Constructors

AddedToken()	Create a new AddedToken object.
AddedToken(String, Boolean)	Create a new AddedToken object from the given content, specifying if it is intended to be a special token. Special tokens are not normalized by default.

Properties

Content

Gets or sets the content of the added token

Methods

Equals(AddedToken)	Determines whether two token instances are equal.
GetHashCode()	Returns the hash code for the current token.

Operators

Implicit(String to AddedToken)

Defines an implicit conversion of a string object to AddedToken.

Applies to

Share via