How to integrate with Log Analytics
Note
We will retire Azure HDInsight on AKS on January 31, 2025. Before January 31, 2025, you will need to migrate your workloads to Microsoft Fabric or an equivalent Azure product to avoid abrupt termination of your workloads. The remaining clusters on your subscription will be stopped and removed from the host.
Only basic support will be available until the retirement date.
Important
This feature is currently in preview. The Supplemental Terms of Use for Microsoft Azure Previews include more legal terms that apply to Azure features that are in beta, in preview, or otherwise not yet released into general availability. For information about this specific preview, see Azure HDInsight on AKS preview information. For questions or feature suggestions, please submit a request on AskHDInsight with the details and follow us for more updates on Azure HDInsight Community.
This article describes how to enable Log Analytics to monitor & collect logs for cluster pool and cluster operations on HDInsight on AKS. You can enable the integration during cluster pool creation or post the creation. Once the integration at cluster pool is enabled, it isn't possible to disable the integration. However, you can disable the log analytics for individual clusters, which are part of the same pool.
Prerequisites
Log Analytics workspace. You can think of this workspace as a unique logs environment with its own data repository, data sources, and solutions. Learn how to create a Log Analytics workspace .
Note
Log Analytics must be enabled at cluster pool level first so as to enable it at a cluster level.
The configuration at cluster pool level is a global switch for all clusters in the cluster pool, therefore all clusters in the same cluster pool can only flow log to one Log Analytics workspace.
Enable Log Analytics using the portal during cluster pool creation
Sign in to Azure portal.
Select Create a resource and search for cluster pool in marketplace and select Azure HDInsight on AKS Cluster Pool. For more information on starting the cluster pool creation process, see Create a cluster pool.
Navigate to the Integrations blade, select Enable Log Analytics.
From the drop-down list, select an existing Log Analytics workspace. Complete the remaining details required to finish cluster pool creation and select Create.
Log Analytics is enabled when the cluster pool is successfully created. All monitoring capabilities can be accessed under your cluster pool’s Monitoring section.
Enable Log Analytics using portal after cluster pool creation
In the Azure portal search bar, type "HDInsight on AKS cluster pools" and select Azure HDInsight on AKS cluster pools to go to the cluster pools page. On the HDInsight on AKS cluster pools page, select your cluster pool.
Navigate to the "Monitor settings" blade on the left side menu and click on "Configure" to enable Log Analytics.
Select an existing Log Analytics workspace, and click Ok.
Enable Log Analytics using the portal during cluster creation
In the Azure portal search bar, type "HDInsight on AKS cluster pools" and select Azure HDInsight on AKS cluster pools to go to the cluster pools page. On the HDInsight on AKS cluster pools page, select the cluster pool in which you want to create a cluster.
Note
It is important to make sure that the selected cluster pool has Log Analytics enabled.
Select New Cluster to start the creation process. For more information on starting the cluster creation process, see Create a cluster.
Navigate to the Integrations blade, select Enable Log Analytics.
Select one or more type of logs you would like to collect. Complete the remaining details required to finish the cluster creation and select Create.
Note
If no option is selected, then only AKS service logs will be available.
Log Analytics is enabled when the cluster is successfully created. All monitoring capabilities can be accessed under your cluster's Monitoring section.
Enable Log Analytics using portal after cluster creation
In the Azure portal top search bar, type "HDInsight on AKS clusters" and select Azure HDInsight on AKS clusters from the drop-down list. On the HDInsight on AKS cluster pools page, select your cluster name from the list page.
Navigate to the "Monitor settings" blade, select Enable Log Analytics. Choose one or more type of logs you would like to collect, and click Save.
Note
If no option is selected, then only AKS service logs will be available.
Access the log tables and run queries using the portal
From the Azure portal, select your cluster pool or cluster of choice to open it.
Navigate to the Monitoring section and select the Logs blade to query and analyze the collected data.
A list of commonly used query templates is provided to choose from to simplify the process or you can write your own query using the provided console.
