Tokenizer.Decode Method

Definition

Namespace:

Microsoft.ML.Tokenizers

Assembly:

Microsoft.ML.Tokenizers.dll

Package:

Microsoft.ML.Tokenizers v0.21.1

Overloads

Decode(IEnumerable<Int32>, Boolean)	Decode the given ids, back to a String.
Decode(Int32, Boolean)	Decodes the Id to the mapped token.

Decode(IEnumerable<Int32>, Boolean)

Decode the given ids, back to a String.

public string? Decode (System.Collections.Generic.IEnumerable<int> ids, bool skipSpecialTokens = false);

member this.Decode : seq<int> * bool -> string

Public Function Decode (ids As IEnumerable(Of Integer), Optional skipSpecialTokens As Boolean = false) As String

Parameters

ids

IEnumerable<Int32>

The list of ids that we want to decode.

skipSpecialTokens

Boolean

Whether the special tokens should be removed from the decoded string.

Returns

String

The decoded string.

Decode(Int32, Boolean)

Decodes the Id to the mapped token.

public string? Decode (int id, bool skipSpecialTokens = false);

member this.Decode : int * bool -> string

Public Function Decode (id As Integer, Optional skipSpecialTokens As Boolean = false) As String

Parameters

id

Int32

The id to map to the token.

skipSpecialTokens

Boolean

Indicate if want to skip the special tokens during the decoding.

Returns

String

The decoded string or null if there is no token mapped to the input id.