RecommendVmSkuTokenBudget interface

Package:

@azure/arm-aiinfraoptimization

The per-request token budget used to size the inference key-value cache.

Properties

maxOutputTokens	The maximum number of generated output tokens per inference request. Must be supplied together with promptTokens, and the two must total from 1 through 131072.
promptTokens	The number of prompt, or input, tokens per inference request. Must be supplied together with maxOutputTokens, and the two must total from 1 through 131072.

Property Details

maxOutputTokens

The maximum number of generated output tokens per inference request. Must be supplied together with promptTokens, and the two must total from 1 through 131072.

maxOutputTokens?: number

Property Value

number

promptTokens

The number of prompt, or input, tokens per inference request. Must be supplied together with maxOutputTokens, and the two must total from 1 through 131072.

promptTokens?: number

Property Value

number