RecommendVmSkuSizingWorkload interface

Package:

@azure/arm-aiinfraoptimization

The resolved per-replica workload inputs used for memory sizing.

Properties

contextLength	The context length, in tokens, used to size the key-value cache: the sum of promptTokens and maxOutputTokens when a token budget is supplied, otherwise the service default.
maxConcurrentRequestsPerReplica	The maximum number of simultaneous inference requests handled by each replica.

Property Details

contextLength

The context length, in tokens, used to size the key-value cache: the sum of promptTokens and maxOutputTokens when a token budget is supplied, otherwise the service default.

contextLength: number

Property Value

number

maxConcurrentRequestsPerReplica

The maximum number of simultaneous inference requests handled by each replica.

maxConcurrentRequestsPerReplica: number

Property Value

number