Edit

Share via

Capacity policy

  • Article

Applies to: ✅ Azure Data Explorer

A capacity policy is used for controlling the compute resources of data management operations on the cluster.

The capacity policy object

The capacity policy is made of the following components:

To view the capacity of your cluster, use the .show capacity command.

Ingestion capacity

Property Type Description
ClusterMaximumConcurrentOperations long The maximum number of concurrent ingestion operations allowed in a cluster. This value caps the total ingestion capacity, as shown in the following formula.
CoreUtilizationCoefficient real Determines the percentage of cores to use in the ingestion capacity calculation.

Formula

The .show capacity command returns the cluster's ingestion capacity based on the following formula:

Minimum(ClusterMaximumConcurrentOperations , Number of nodes in cluster * Maximum(1, Core count per node * CoreUtilizationCoefficient))

Note

In clusters with four or more nodes, the admin node doesn't participate in ingestion operations, so the Number of nodes in cluster is reduced by one.

Extents merge capacity

Property Type Description
MinimumConcurrentOperationsPerNode long The minimal number of concurrent extents merge/rebuild operations on a single node. Default is 1.
MaximumConcurrentOperationsPerNode long The maximum number of concurrent extents merge/rebuild operations on a single node. Default is 5.

Formula

The .show capacity command returns the cluster's extents merge capacity based on the following formula:

Number of nodes in cluster * Concurrent operations per node

The effective value for Concurrent operations per node is automatically adjusted by the system in the range [MinimumConcurrentOperationsPerNode,MaximumConcurrentOperationsPerNode], as long as the success rate of the merge operations is 90% or higher.

Note

In clusters with four or more nodes, the admin node doesn't participate in merge operations, so Number of nodes in cluster is reduced by one.

Extents purge rebuild capacity

Property Type Description
MaximumConcurrentOperationsPerNode long The maximum number of concurrent rebuild extents for purge operations on a single node.

Formula

The .show capacity command returns the cluster's extents purge rebuild capacity based on the following formula:

Number of nodes in cluster x MaximumConcurrentOperationsPerNode

Note

In clusters with four or more nodes, the admin node doesn't participate in purge operations, so Number of nodes in cluster is reduced by one.

Export capacity

Property Type Description
ClusterMaximumConcurrentOperations long The maximum number of concurrent export operations in a cluster. This value caps the total export capacity, as shown in the following formula.
CoreUtilizationCoefficient long Determines the percentage of cores to use in the export capacity calculation.

Formula

The .show capacity command returns the cluster's export capacity based on the following formula:

Minimum(ClusterMaximumConcurrentOperations , Number of nodes in cluster * Maximum(1, Core count per node * CoreUtilizationCoefficient))

Note

In clusters with four or more nodes, the admin node doesn't participate in export operations. The Number of nodes in cluster is reduced by one.

Extents partition capacity

Property Type Description
ClusterMinimumConcurrentOperations long The minimal number of concurrent extents partition operations in a cluster. Default is 1.
ClusterMaximumConcurrentOperations long The maximum number of concurrent extents partition operations in a cluster. Default is 32.

The effective value for Concurrent operations is automatically adjusted by the system in the range [ClusterMinimumConcurrentOperations,ClusterMaximumConcurrentOperations], as long as the success rate of the partitioning operations is 90% or higher.

Materialized views capacity policy

The policy can be used to change concurrency settings for materialized views. Changing the materialized views capacity policy can be useful when there's more than a single materialized view defined on a cluster.

Property Type Description
ClusterMinimumConcurrentOperations long The minimal number of concurrent materialization operations in a cluster. Default is 1.
ClusterMaximumConcurrentOperations long The maximum number of concurrent materialization operations in a cluster. Default is 10.

By default, only a single materialization runs concurrently (see how materialized views work). The system adjusts the current concurrency in the range [ClusterMinimumConcurrentOperations,ClusterMaximumConcurrentOperations], based on the number of materialized views in the cluster and the cluster's CPU. You can increase/decrease concurrency by altering this policy. For example, if the cluster has ten materialized views, setting the ClusterMinimumConcurrentOperations to five ensures that at least five of them can materialize concurrently. You can view the effective value for the current concurrency using the .show capacity command

Warning

Raising the ClusterMinimumConcurrentOperations can lead to resource exhaustion and degrade cluster performance. Carefully monitor the cluster's health and increase concurrency gradually if you manually alter this policy.

Stored query results capacity

Property Type Description
MaximumConcurrentOperationsPerDbAdmin long The maximum number of concurrent ingestion operations in a cluster admin node.
CoreUtilizationCoefficient real Determines the percentage of cores to use in the stored query results creation calculation.

Formula

The .show capacity command returns the cluster's stored query results creation capacity based on the following formula:

Number of nodes in cluster * Maximum(1, Core count per node * CoreUtilizationCoefficient)

Note

In clusters with four or more nodes, the admin node doesn't participate in stored query results creation operations, so the Number of nodes in cluster is reduced by one.

Streaming ingestion post processing capacity

Property Type Description
MaximumConcurrentOperationsPerNode long The maximum number of concurrent streaming ingestion post processing operations on each cluster node.

Formula

The .show capacity command returns the cluster's streaming ingestion post processing capacity based on the following formula:

Number of nodes in cluster x MaximumConcurrentOperationsPerNode

Note

In clusters with four or more nodes, the admin node doesn't participate in streaming ingestion post processing, so Number of nodes in cluster is reduced by one.

Purge storage artifacts cleanup capacity

Property Type Description
MaximumConcurrentOperationsPerCluster long The maximum number of concurrent purge storage artifacts cleanup operations on cluster.

Formula

The .show capacity command returns the cluster's purge storage artifacts cleanup capacity based on the following formula:

MaximumConcurrentOperationsPerCluster

Periodic storage artifacts cleanup capacity

Property Type Description
MaximumConcurrentOperationsPerCluster long The maximum number of concurrent periodic storage artifacts cleanup operations on cluster.

Formula

The .show capacity command returns the cluster's periodic storage artifacts cleanup capacity based on the following formula:

MaximumConcurrentOperationsPerCluster

Defaults

The default capacity policy has the following JSON representation:

{
  "IngestionCapacity": {
    "ClusterMaximumConcurrentOperations": 512,
    "CoreUtilizationCoefficient": 0.75
  },
  "ExtentsMergeCapacity": {
    "MinimumConcurrentOperationsPerNode": 1,
    "MaximumConcurrentOperationsPerNode": 3
  },
  "ExtentsPurgeRebuildCapacity": {
    "MaximumConcurrentOperationsPerNode": 1
  },
  "ExportCapacity": {
    "ClusterMaximumConcurrentOperations": 100,
    "CoreUtilizationCoefficient": 0.25
  },
  "ExtentsPartitionCapacity": {
    "ClusterMinimumConcurrentOperations": 1,
    "ClusterMaximumConcurrentOperations": 32
  },
  "MaterializedViewsCapacity": {
    "ClusterMaximumConcurrentOperations": 1,
    "ExtentsRebuildCapacity": {
      "ClusterMaximumConcurrentOperations": 50,
      "MaximumConcurrentOperationsPerNode": 5
    }
  },
  "StoredQueryResultsCapacity": {
    "MaximumConcurrentOperationsPerDbAdmin": 250,
    "CoreUtilizationCoefficient": 0.75
  },
  "StreamingIngestionPostProcessingCapacity": {
    "MaximumConcurrentOperationsPerNode": 4
  },
  "PurgeStorageArtifactsCleanupCapacity": {
    "MaximumConcurrentOperationsPerCluster": 2
  },
  "PeriodicStorageArtifactsCleanupCapacity": {
    "MaximumConcurrentOperationsPerCluster": 2
  }
}

Management commands

Warning

Consult with the support team before altering a capacity policy.

Management commands throttling

Kusto limits the number of concurrent requests for the following user-initiated commands:

  • Ingestions
  • Purges
    • The global limit is currently fixed at one per cluster.
    • The purge rebuild capacity is used internally to determine the number of concurrent rebuild operations during purge commands. Purge commands won't be blocked or throttled because of this process, but will complete faster or slower depending on the purge rebuild capacity.
  • Exports

When the cluster detects that an operation has exceeded the limit on concurrent requests:

  • The command's state, as presented by System information commands, is Throttled.
  • The error message includes the command type, the origin of the throttling and the capacity that's been exceeded. For example:
    • For example: The management command was aborted due to throttling. Retrying after some backoff might succeed. CommandType: 'TableSetOrAppend', Capacity: 18, Origin: 'CapacityPolicy/Ingestion'.
  • The HTTP response code is 429. The subcode is TooManyRequests.
  • The exception type is ControlCommandThrottledException.

Note

Management commands may also be throttled as a result of exceeding the limit defined by a workload group's Request rate limit policy.

Related content