AzureOpenAITokenizerParameters Class

Definition

The AzureOpenAITokenizerParameters.

public class AzureOpenAITokenizerParameters
type AzureOpenAITokenizerParameters = class
Public Class AzureOpenAITokenizerParameters
Inheritance
AzureOpenAITokenizerParameters

Constructors

AzureOpenAITokenizerParameters()

Initializes a new instance of AzureOpenAITokenizerParameters.

Properties

AllowedSpecialTokens

(Optional) Only applies if the unit is set to azureOpenAITokens. This parameter defines a collection of special tokens that are permitted within the tokenization process.

EncoderModelName

Only applies if the unit is set to azureOpenAITokens. Options include 'R50k_base', 'P50k_base', 'P50k_edit' and 'CL100k_base'. The default value is 'CL100k_base'.

Applies to