RecommendVmSkuWorkload interface

Package:

@azure/arm-aiinfraoptimization

Inference workload assumptions that affect memory sizing.

Properties

maxConcurrentRequestsPerReplica	The maximum number of simultaneous inference requests handled by each deployment replica.
tokenBudget	The prompt and output token budget used for key-value cache sizing.

Property Details

maxConcurrentRequestsPerReplica

The maximum number of simultaneous inference requests handled by each deployment replica.

maxConcurrentRequestsPerReplica?: number

Property Value

number

tokenBudget

The prompt and output token budget used for key-value cache sizing.

tokenBudget?: RecommendVmSkuTokenBudget

Property Value

RecommendVmSkuTokenBudget