Capacity policy
Applies to: ✅ Azure Data Explorer
A capacity policy is used for controlling the compute resources of data management operations on the cluster.
The capacity policy object
The capacity policy is made of the following components:
- IngestionCapacity
- ExtentsMergeCapacity
- ExtentsPurgeRebuildCapacity
- ExportCapacity
- ExtentsPartitionCapacity
- MaterializedViewsCapacity
- StoredQueryResultsCapacity
- StreamingIngestionPostProcessingCapacity
- PurgeStorageArtifactsCleanupCapacity
- PeriodicStorageArtifactsCleanupCapacity
To view the capacity of your cluster, use the .show capacity command.
Ingestion capacity
Property | Type | Description |
---|---|---|
ClusterMaximumConcurrentOperations |
long |
The maximum number of concurrent ingestion operations allowed in a cluster. This value caps the total ingestion capacity, as shown in the following formula. |
CoreUtilizationCoefficient |
real |
Determines the percentage of cores to use in the ingestion capacity calculation. |
Formula
The .show capacity command returns the cluster's ingestion capacity based on the following formula:
Minimum(ClusterMaximumConcurrentOperations
,
Number of nodes in cluster *
Maximum(1,
Core count per node *
CoreUtilizationCoefficient))
Note
In clusters with four or more nodes, the admin node doesn't participate in ingestion operations, so the Number of nodes in cluster is reduced by one.
Extents merge capacity
Property | Type | Description |
---|---|---|
MinimumConcurrentOperationsPerNode |
long |
The minimal number of concurrent extents merge/rebuild operations on a single node. Default is 1 . |
MaximumConcurrentOperationsPerNode |
long |
The maximum number of concurrent extents merge/rebuild operations on a single node. Default is 5 . |
Formula
The .show capacity command returns the cluster's extents merge capacity based on the following formula:
Number of nodes in cluster *
Concurrent operations per node
The effective value for Concurrent operations per node is automatically adjusted by the system in the range [MinimumConcurrentOperationsPerNode
,MaximumConcurrentOperationsPerNode
], as long as the success rate of the merge operations is 90% or higher.
Note
In clusters with four or more nodes, the admin node doesn't participate in merge operations, so Number of nodes in cluster is reduced by one.
Extents purge rebuild capacity
Property | Type | Description |
---|---|---|
MaximumConcurrentOperationsPerNode |
long |
The maximum number of concurrent rebuild extents for purge operations on a single node. |
Formula
The .show capacity command returns the cluster's extents purge rebuild capacity based on the following formula:
Number of nodes in cluster x MaximumConcurrentOperationsPerNode
Note
In clusters with four or more nodes, the admin node doesn't participate in purge operations, so Number of nodes in cluster is reduced by one.
Export capacity
Property | Type | Description |
---|---|---|
ClusterMaximumConcurrentOperations |
long |
The maximum number of concurrent export operations in a cluster. This value caps the total export capacity, as shown in the following formula. |
CoreUtilizationCoefficient |
long |
Determines the percentage of cores to use in the export capacity calculation. |
Formula
The .show capacity command returns the cluster's export capacity based on the following formula:
Minimum(ClusterMaximumConcurrentOperations
,
Number of nodes in cluster *
Maximum(1,
Core count per node *
CoreUtilizationCoefficient))
Note
In clusters with four or more nodes, the admin node doesn't participate in export operations. The Number of nodes in cluster
is reduced by one.
Extents partition capacity
Property | Type | Description |
---|---|---|
ClusterMinimumConcurrentOperations |
long |
The minimal number of concurrent extents partition operations in a cluster. Default is 1 . |
ClusterMaximumConcurrentOperations |
long |
The maximum number of concurrent extents partition operations in a cluster. Default is 32 . |
The effective value for Concurrent operations is automatically adjusted by the system in the range
[ClusterMinimumConcurrentOperations
,ClusterMaximumConcurrentOperations
], as long as the success rate of the
partitioning operations is 90% or higher.
Materialized views capacity policy
The policy can be used to change concurrency settings for materialized views. Changing the materialized views capacity policy can be useful when there's more than a single materialized view defined on a cluster.
Property | Type | Description |
---|---|---|
ClusterMinimumConcurrentOperations |
long |
The minimal number of concurrent materialization operations in a cluster. Default is 1 . |
ClusterMaximumConcurrentOperations |
long |
The maximum number of concurrent materialization operations in a cluster. Default is 10 . |
The effective value for Concurrent operations is automatically adjusted by the system in the range [ClusterMinimumConcurrentOperations
,ClusterMaximumConcurrentOperations
], based on the number of materialized views in the cluster and the cluster's CPU.
Warning
Only increase ClusterMinimumConcurrentOperations
if the cluster has ample resources (low CPU usage and available memory). Raising these values under resource constraints can lead to exhaustion and significantly degrade cluster performance.
Stored query results capacity
Property | Type | Description |
---|---|---|
MaximumConcurrentOperationsPerDbAdmin |
long |
The maximum number of concurrent ingestion operations in a cluster admin node. |
CoreUtilizationCoefficient |
real |
Determines the percentage of cores to use in the stored query results creation calculation. |
Formula
The .show capacity command returns the cluster's stored query results creation capacity based on the following formula:
Number of nodes in cluster *
Maximum(1,
Core count per node *
CoreUtilizationCoefficient)
Note
In clusters with four or more nodes, the admin node doesn't participate in stored query results creation operations, so the Number of nodes in cluster is reduced by one.
Streaming ingestion post processing capacity
Property | Type | Description |
---|---|---|
MaximumConcurrentOperationsPerNode |
long |
The maximum number of concurrent streaming ingestion post processing operations on each cluster node. |
Formula
The .show capacity command returns the cluster's streaming ingestion post processing capacity based on the following formula:
Number of nodes in cluster x MaximumConcurrentOperationsPerNode
Note
In clusters with four or more nodes, the admin node doesn't participate in streaming ingestion post processing, so Number of nodes in cluster is reduced by one.
Purge storage artifacts cleanup capacity
Property | Type | Description |
---|---|---|
MaximumConcurrentOperationsPerCluster |
long |
The maximum number of concurrent purge storage artifacts cleanup operations on cluster. |
Formula
The .show capacity command returns the cluster's purge storage artifacts cleanup capacity based on the following formula:
MaximumConcurrentOperationsPerCluster
Periodic storage artifacts cleanup capacity
Property | Type | Description |
---|---|---|
MaximumConcurrentOperationsPerCluster |
long |
The maximum number of concurrent periodic storage artifacts cleanup operations on cluster. |
Formula
The .show capacity command returns the cluster's periodic storage artifacts cleanup capacity based on the following formula:
MaximumConcurrentOperationsPerCluster
Defaults
The default capacity policy has the following JSON representation:
{
"IngestionCapacity": {
"ClusterMaximumConcurrentOperations": 512,
"CoreUtilizationCoefficient": 0.75
},
"ExtentsMergeCapacity": {
"MinimumConcurrentOperationsPerNode": 1,
"MaximumConcurrentOperationsPerNode": 3
},
"ExtentsPurgeRebuildCapacity": {
"MaximumConcurrentOperationsPerNode": 1
},
"ExportCapacity": {
"ClusterMaximumConcurrentOperations": 100,
"CoreUtilizationCoefficient": 0.25
},
"ExtentsPartitionCapacity": {
"ClusterMinimumConcurrentOperations": 1,
"ClusterMaximumConcurrentOperations": 32
},
"MaterializedViewsCapacity": {
"ClusterMaximumConcurrentOperations": 1,
"ExtentsRebuildCapacity": {
"ClusterMaximumConcurrentOperations": 50,
"MaximumConcurrentOperationsPerNode": 5
}
},
"StoredQueryResultsCapacity": {
"MaximumConcurrentOperationsPerDbAdmin": 250,
"CoreUtilizationCoefficient": 0.75
},
"StreamingIngestionPostProcessingCapacity": {
"MaximumConcurrentOperationsPerNode": 4
},
"PurgeStorageArtifactsCleanupCapacity": {
"MaximumConcurrentOperationsPerCluster": 2
},
"PeriodicStorageArtifactsCleanupCapacity": {
"MaximumConcurrentOperationsPerCluster": 2
}
}
Management commands
Warning
Consult with the support team before altering a capacity policy.
- Use
.show cluster policy capacity
to show the current capacity policy of the cluster. - Use
.alter-merge cluster policy capacity
to alter the capacity policy of the cluster.
Management commands throttling
Kusto limits the number of concurrent requests for the following user-initiated commands:
- Ingestions
- This category includes commands that ingest from storage, ingest from a query, and ingest inline.
- The limit is as defined by the ingestion capacity.
- Purges
- The global limit is currently fixed at one per cluster.
- The purge rebuild capacity is used internally to determine the number of concurrent rebuild operations during purge commands. Purge commands won't be blocked or throttled because of this process, but will complete faster or slower depending on the purge rebuild capacity.
- Exports
- The limit is as defined in the export capacity.
When the cluster detects that an operation has exceeded the limit on concurrent requests:
- The command's state, as presented by System information commands, will be
Throttled
. - The error message will include the command type, the origin of the throttling and the capacity that's been exceeded. For example:
- For example:
The management command was aborted due to throttling. Retrying after some backoff might succeed. CommandType: 'TableSetOrAppend', Capacity: 18, Origin: 'CapacityPolicy/Ingestion'
.
- For example:
- The HTTP response code will be
429
. The subcode will beTooManyRequests
. - The exception type will be
ControlCommandThrottledException
.
Note
Management commands may also be throttled as a result of exceeding the limit defined by a workload group's Request rate limit policy.