Tokenizer interface

Reference

Package:: @microsoft/teams-ai

Copyright (c) Microsoft Corporation. All rights reserved. Licensed under the MIT License.

Methods

decode(number[])
encode(string)

Method Details

decode(number[])

function decode(tokens: number[]): string

Parameters

tokens: number[]

Returns

string

encode(string)

function encode(text: string): number[]

Parameters

text: string

Returns

number[]