Edit

Create and manage a dedicated cluster in Azure Monitor Logs

  • Article

Linking a Log Analytics workspace to a dedicated cluster in Azure Monitor provides advanced capabilities and higher query utilization. Clusters require a minimum ingestion commitment of 100 GB per day. You can link and unlink workspaces from a dedicated cluster without any data loss or service interruption.

Advanced capabilities

Capabilities that require dedicated clusters:

  • Customer-managed keys - Encrypt cluster data using keys that you provide and control.
  • Lockbox - Control Microsoft support engineer access requests to your data.
  • Double encryption - Protect against a scenario where one of the encryption algorithms or keys may be compromised. In this case, the extra layer of encryption continues to protect your data.
  • Cross-query optimization - Cross-workspace queries run faster when workspaces are on the same cluster.
  • Cost optimization - Link your workspaces in same region to cluster to get commitment tier discount to all workspaces, even to ones with low ingestion that eligible for commitment tier discount.
  • Availability zones - Protect your data from datacenter failures by relying on datacenters in different physical locations, equipped with independent power, cooling, and networking. The physical separation in zones and independent infrastructure makes an incident far less likely since the workspace can rely on the resources from any of the zones. Azure Monitor availability zones covers broader parts of the service and when available in your region, extends your Azure Monitor resilience automatically. Azure Monitor creates dedicated clusters as availability-zone-enabled (isAvailabilityZonesEnabled: 'true') by default in supported regions. Dedicated clusters Availability zones aren't supported in all regions currently.
  • Ingest from Azure Event Hubs - Lets you ingest data directly from an event hub into a Log Analytics workspace. Dedicated cluster lets you use capability when ingestion from all linked workspaces combined meet commitment tier.

Cluster pricing model

Log Analytics Dedicated Clusters use a commitment tier pricing model of at least 100 GB/day. Any usage above the tier level incurs charges based on the per-GB rate of that commitment tier. See Azure Monitor Logs pricing details for pricing details for dedicated clusters. The commitment tiers have a 31-day commitment period from the time a commitment tier is selected.

Prerequisites

  • Dedicated clusters require a minimum ingestion commitment of 100 GB per day.
  • When creating a dedicated cluster, you can't name it with the same name as a cluster that was deleted within the past two weeks.

Required permissions

To perform cluster-related actions, you need these permissions:

Action Permissions or role needed
Create a dedicated cluster Microsoft.Resources/deployments/*and Microsoft.OperationalInsights/clusters/write permissions, as provided by the Log Analytics Contributor built-in role, for example
Change cluster properties Microsoft.OperationalInsights/clusters/write permissions, as provided by the Log Analytics Contributor built-in role, for example
Link workspaces to a cluster Microsoft.OperationalInsights/clusters/write, Microsoft.OperationalInsights/workspaces/write, and Microsoft.OperationalInsights/workspaces/linkedservices/write permissions, as provided by the Log Analytics Contributor built-in role, for example
Check workspace link status Microsoft.OperationalInsights/workspaces/read permissions to the workspace, as provided by the Log Analytics Reader built-in role, for example
Get clusters or check a cluster's provisioning status Microsoft.OperationalInsights/clusters/read permissions, as provided by the Log Analytics Reader built-in role, for example
Update commitment tier or billingType in a cluster Microsoft.OperationalInsights/clusters/write permissions, as provided by the Log Analytics Contributor built-in role, for example
Grant the required permissions Owner or Contributor role that has */write permissions, or the Log Analytics Contributor built-in role, which has Microsoft.OperationalInsights/* permissions
Unlink a workspace from cluster Microsoft.OperationalInsights/workspaces/linkedServices/delete permissions, as provided by the Log Analytics Contributor built-in role, for example
Delete a dedicated cluster Microsoft.OperationalInsights/clusters/delete permissions, as provided by the Log Analytics Contributor built-in role, for example

For more information on Log Analytics permissions, see Manage access to log data and workspaces in Azure Monitor.

Create a dedicated cluster

Provide the following properties when creating new dedicated cluster:

  • ClusterName: Must be unique for the resource group.
  • ResourceGroupName: Use a central IT resource group because many teams in the organization usually share clusters. For more design considerations, review Design a Log Analytics workspace configuration.
  • Location
  • SkuCapacity: You can set the commitment tier to 100, 200, 300, 400, 500, 1000, 2000, 5000, 10000, 25000, 50000 GB per day. The minimum commitment tier supported in CLI is 500 currently. Use REST to configure lower commitment tiers with minimum of 100. For more information on cluster costs, see Dedicate clusters.
  • Managed identity: Clusters support two managed identity types:

    • System-assigned managed identity - Generated automatically with the cluster creation when identity type is set to "SystemAssigned". This identity can be used later to grant storage access to your Key Vault for wrap and unwrap operations.

      Identity in Cluster's REST Call

      {
  "identity": {
    "type": "SystemAssigned"
    }
}

    • User-assigned managed identity - Lets you configure a customer-managed key at cluster creation, when granting it permissions in your Key Vault before cluster creation.

      Identity in Cluster's REST Call

      {
"identity": {
  "type": "UserAssigned",
    "userAssignedIdentities": {
      "subscriptions/<subscription-id>/resourcegroups/<resource-group-name>/providers/Microsoft.ManagedIdentity/UserAssignedIdentities/<cluster-assigned-managed-identity>"
    }
  }  
}

After you create your cluster resource, you can edit properties such as sku, *keyVaultProperties, or billingType. See more details below.

Deleted clusters take two weeks to be completely removed. You can have up to seven clusters per subscription and region, five active, and two deleted in past two weeks.

Note

Cluster creation triggers resource allocation and provisioning. This operation can take a few hours to complete. Dedicated cluster is billed once provisioned regardless data ingestion and it's recommended to prepare the deployment to expedite the provisioning and workspaces link to cluster. Verify the following:

  • A list of initial workspace to be linked to cluster is identified
  • You have permissions to subscription intended for the cluster and any workspace to be linked

N/A

Check cluster provisioning status

The provisioning of the Log Analytics cluster takes a while to complete. Use one of the following methods to check the ProvisioningState property. The value is ProvisioningAccount while provisioning and Succeeded when completed.

N/A

Note

  • Linking a workspace can be performed only after the completion of the Log Analytics cluster provisioning.
  • Linking a workspace to a cluster involves syncing multiple backend components and cache hydration, which can take up to two hours.
  • When linking a Log Analytics workspace workspace, the workspace billing plan in changed to LACluster, and you should remove sku in workspace template to prevent conflict during workspace deployment.
  • Other than the billing aspects that is governed by the cluster plan, all workspace configurations and query aspects remain unchanged during and after the link.

You need 'write' permissions to both the workspace and the cluster resource for workspace link operation:

  • In the workspace: Microsoft.OperationalInsights/workspaces/write
  • In the cluster resource: Microsoft.OperationalInsights/clusters/write

Once Log Analytics workspace linked to a dedicated cluster, new data sent to workspace is ingested to your dedicated cluster, while previously ingested data remains in Log Analytics cluster. Linking a workspace has no effect on workspace operation, including ingestion and query experiences. Log Analytics query engine stitches data from old and new clusters automatically, and the results of queries are complete.

When dedicated cluster is configured with customer-managed key (CMK), new ingested data is encrypted with your key, while older data remains encrypted with Microsoft-managed key (MMK). The key configuration is abstracted by Log Analytics and the query across old and new data encryptions is performed seamlessly.

A cluster can be linked to up to 1,000 workspaces located in the same region with cluster. A workspace can't be linked to a cluster more than twice a month, to prevent data fragmentation.

The workspace and the cluster can be in different subscriptions. It's possible for the workspace and cluster to be in different tenants if Azure Lighthouse is used to map both of them to a single tenant.

Use the following steps to link a workspace to a cluster. You can use automation for linking multiple workspaces:

N/A

When a cluster is configured with customer-managed keys, data ingested to the workspaces after the link operation completion is stored encrypted with your managed key. The workspace link operation can take up to 90 minutes to complete and you can check the state by sending Get request to workspace and observe if clusterResourceId property is present in the response under features.

  1. Open the Log Analytics workspaces menu and then select your workspace.
  2. On the Overview page, select JSON View.

Change cluster properties

After you create your cluster resource and it's fully provisioned, you can edit cluster properties using CLI, PowerShell or REST API. Properties you can set after the cluster is provisioned include:

  • keyVaultProperties - Contains the key in Azure Key Vault with the following parameters: KeyVaultUri, KeyName, KeyVersion. See Update cluster with Key identifier details.
  • Identity - The identity used to authenticate to your Key Vault. This can be System-assigned or User-assigned.
  • billingType - Billing attribution for the cluster resource and its data. Includes on the following values:
    • Cluster (default)--The costs for your cluster are attributed to the cluster resource.
    • Workspaces--The costs for your cluster are attributed proportionately to the workspaces in the Cluster, with the cluster resource being billed some of the usage if the total ingested data for the day is under the commitment tier. See Log Analytics Dedicated Clusters to learn more about the cluster pricing model.

Important

Cluster update should not include both identity and key identifier details in the same operation. If you need to update both, the update should be in two consecutive operations.

Note

The billingType property isn't supported in CLI.

Get all clusters in resource group

N/A

Get all clusters in subscription

N/A

Update commitment tier in cluster

When the data volume to linked workspaces changes over time, you can update the Commitment Tier level appropriately to optimize cost. The tier is specified in units of Gigabytes (GB) and can have values of 100, 200, 300, 400, 500, 1000, 2000, 5000, 10000, 25000, 50000 GB per day. You don't have to provide the full REST request body, but you must include the sku.

During the commitment period, you can change to a higher commitment tier, which restarts the 31-day commitment period. You can't move back to pay-as-you-go or to a lower commitment tier until after you finish the commitment period.

N/A

Update billingType in cluster

The billingType property determines the billing attribution for the cluster and its data:

  • Cluster (default) -- billing is attributed to the Cluster resource
  • Workspaces -- billing is attributed to linked workspaces proportionally. When data volume from all linked workspaces is below Commitment Tier level, the bill for the remaining volume is attributed to the cluster

N/A

You can unlink a workspace from a cluster at any time. The workspace pricing tier is changed to per-GB, data ingested to cluster before the unlink operation remains in the cluster, and new data to workspace get ingested to Log Analytics.

Warning

Unlinking a workspace does not move workspace data out of the cluster. Any data collected for workspace while linked to cluster, remains in cluster for the retention period defined in workspace, and accessible as long as cluster isn't deleted.

Queries aren't affected when workspace is unlinked and service performs cross-cluster queries seamlessly. If cluster was configured with Customer-managed key (CMK), data ingested to workspace while was linked, remains encrypted with your key and accessible, while your key and permissions to Key Vault remain.

Note

  • There is a limit of two link operations for a specific workspace within a month to prevent data distribution across clusters. Contact support if you reach the limit.
  • Unlinked workspaces are moved to Pay-As-You-Go pricing tier.

Use the following commands to unlink a workspace from cluster:

N/A

Delete cluster

You need to have write permissions on the cluster resource.

Cluster deletion operation should be done with caution, since operation is non-recoverable. All ingested data to cluster from linked workspaces, gets permanently deleted.

The cluster's billing stops when cluster is deleted, regardless of the 31-days commitment tier defined in cluster.

If you delete a cluster that has linked workspaces, workspaces get automatically unlinked from the cluster, workspaces are moved to pay-as-you-go pricing tier, and new data to workspaces is ingested to Log Analytics clusters instead. You can query workspace for the time range before it was linked to the cluster, and after the unlink, and the service performs cross-cluster queries seamlessly.

Note

  • There is a limit of seven clusters per subscription and region, five active, plus two that were deleted in past two weeks.
  • Cluster's name remain reserved two weeks after deletion, and can't be used for creating a new cluster.

Use the following commands to delete a cluster:

N/A

Limits and constraints

  • A maximum of five active clusters can be created in each region and subscription.

  • A maximum of seven clusters allowed per subscription and region, five active, plus two that were deleted in past 2 weeks.

  • A maximum of 1,000 Log Analytics workspaces can be linked to a cluster.

  • A maximum of two workspace link operations on particular workspace is allowed in 30 day period.

  • Moving a cluster to another resource group or subscription isn't currently supported.

  • Moving a cluster to another region isn't supported.

  • Cluster update shouldn't include both identity and key identifier details in the same operation. In case you need to update both, the update should be in two consecutive operations.

  • Lockbox isn't currently available in China.

  • Double encryption is configured automatically for clusters created from October 2020 in supported regions. You can verify if your cluster is configured for double encryption by sending a GET request on the cluster and observing that the isDoubleEncryptionEnabled value is true for clusters with Double encryption enabled.

    • If you create a cluster and get an error "region-name doesn't support Double Encryption for clusters.", you can still create the cluster without Double encryption by adding "properties": {"isDoubleEncryptionEnabled": false} in the REST request body.
    • Double encryption setting can't be changed after the cluster has been created.

  • Deleting a workspace is permitted while linked to cluster. If you decide to recover the workspace during the soft-delete period, workspace returns to previous state and remains linked to cluster.

  • During the commitment period, you can change to a higher commitment tier, which restarts the 31-day commitment period. You can't move back to pay-as-you-go or to a lower commitment tier until after you finish the commitment period.

Troubleshooting

  • If you get conflict error when creating a cluster, it might have been deleted in past 2 weeks and in deletion process yet. The cluster name remains reserved during the 2 weeks deletion period and you can't create a new cluster with that name.

  • If you update your cluster while the cluster is at provisioning or updating state, the update will fail.

  • Some operations are long and can take a while to complete. These are cluster create, cluster key update and cluster delete. You can check the operation status by sending GET request to cluster or workspace and observe the response. For example, unlinked workspace won't have the clusterResourceId under features.

  • If you attempt to link a Log Analytics workspace that's already linked to another cluster, the operation will fail.

Error messages

Cluster Create

  • 400--Cluster name isn't valid. Cluster name can contain characters a-z, A-Z, 0-9 and length of 3-63.
  • 400--The body of the request is null or in bad format.
  • 400--SKU name is invalid. Set SKU name to capacityReservation.
  • 400--Capacity was provided but SKU isn't capacityReservation. Set SKU name to capacityReservation.
  • 400--Missing Capacity in SKU. Set Capacity value to 100, 200, 300, 400, 500, 1000, 2000, 5000, 10000, 25000, 50000 GB per day.
  • 400--Capacity is locked for 30 days. Decreasing capacity is permitted 30 days after update.
  • 400--No SKU was set. Set the SKU name to capacityReservation and Capacity value to 100, 200, 300, 400, 500, 1000, 2000, 5000, 10000, 25000, 50000 GB per day.
  • 400--Identity is null or empty. Set Identity with systemAssigned type.
  • 400--KeyVaultProperties are set on creation. Update KeyVaultProperties after cluster creation.
  • 400--Operation can't be executed now. Async operation is in a state other than succeeded. Cluster must complete its operation before any update operation is performed.

Cluster Update

  • 400--Cluster is in deleting state. Async operation is in progress. Cluster must complete its operation before any update operation is performed.
  • 400--KeyVaultProperties isn't empty but has a bad format. See key identifier update.
  • 400--Failed to validate key in Key Vault. Could be due to lack of permissions or when key doesn't exist. Verify that you set key and access policy in Key Vault.
  • 400--Key isn't recoverable. Key Vault must be set to Soft-delete and Purge-protection. See Key Vault documentation
  • 400--Operation can't be executed now. Wait for the Async operation to complete and try again.
  • 400--Cluster is in deleting state. Wait for the Async operation to complete and try again.

Cluster Get

  • 404--Cluster not found, the cluster might have been deleted. If you try to create a cluster with that name and get conflict, the cluster is in deletion process.

Cluster Delete

  • 409--Can't delete a cluster while in provisioning state. Wait for the Async operation to complete and try again.
  • 404--Workspace not found. The workspace you specified doesn't exist or was deleted.
  • 409--Workspace link or unlink operation in process.
  • 400--Cluster not found, the cluster you specified doesn't exist or was deleted.
  • 404--Workspace not found. The workspace you specified doesn't exist or was deleted.
  • 409--Workspace link or unlink operation in process.

Next steps