Edit

How to integrate with Log Analytics

  • Article

Important

This feature is currently in preview. The Supplemental Terms of Use for Microsoft Azure Previews include more legal terms that apply to Azure features that are in beta, in preview, or otherwise not yet released into general availability. For information about this specific preview, see Azure HDInsight on AKS preview information. For questions or feature suggestions, please submit a request on AskHDInsight with the details and follow us for more updates on Azure HDInsight Community.

This article describes how to enable Log Analytics to monitor & collect logs for cluster pool and cluster operations on HDInsight on AKS. You can enable the integration during cluster pool creation or post the creation. Once the integration at cluster pool is enabled, it isn't possible to disable the integration. However, you can disable the log analytics for individual clusters, which are part of the same pool.

Prerequisites

  • Log Analytics workspace. You can think of this workspace as a unique logs environment with its own data repository, data sources, and solutions. Learn how to create a Log Analytics workspace .

    Note

    1. Log Analytics must be enabled at cluster pool level first so as to enable it at a cluster level.

    2. The configuration at cluster pool level is a global switch for all clusters in the cluster pool, therefore all clusters in the same cluster pool can only flow log to one Log Analytics workspace.

Enable Log Analytics using the portal during cluster pool creation

  1. Sign in to Azure portal.

  2. Select Create a resource and search for cluster pool in marketplace and select Azure HDInsight on AKS Cluster Pool. For more information on starting the cluster pool creation process, see Create a cluster pool.

  3. Navigate to the Integrations blade, select Enable Log Analytics.

    Screenshot showing how to enable log analytics option.

  4. From the drop-down list, select an existing Log Analytics workspace. Complete the remaining details required to finish cluster pool creation and select Create.

  5. Log Analytics is enabled when the cluster pool is successfully created. All monitoring capabilities can be accessed under your cluster pool’s Monitoring section.

    Screenshot showing monitoring section in the Azure portal.

Enable Log Analytics using portal after cluster pool creation

  1. In the Azure portal search bar, type "HDInsight on AKS cluster pools" and select Azure HDInsight on AKS cluster pools to go to the cluster pools page. On the HDInsight on AKS cluster pools page, select your cluster pool.

    Screenshot showing search option for getting started with HDInsight on AKS Cluster pool.

    Screenshot showing cluster pools in a list view.

  2. Navigate to the "Monitor settings" blade on the left side menu and click on "Configure" to enable Log Analytics.

    Screenshot showing cluster pool integration blade.

  3. Select an existing Log Analytics workspace, and click Ok.

    Screenshot showing how to enable cluster pool log analytics.

Enable Log Analytics using the portal during cluster creation

  1. In the Azure portal search bar, type "HDInsight on AKS cluster pools" and select Azure HDInsight on AKS cluster pools to go to the cluster pools page. On the HDInsight on AKS cluster pools page, select the cluster pool in which you want to create a cluster.

    Screenshot showing search option for getting started with HDInsight on AKS Cluster pool.

    Screenshot showing cluster pools in a list view.

    Note

    It is important to make sure that the selected cluster pool has Log Analytics enabled.

  2. Select New Cluster to start the creation process. For more information on starting the cluster creation process, see Create a cluster.

    Screenshot showing New cluster button in the Azure portal.

  3. Navigate to the Integrations blade, select Enable Log Analytics.

  4. Select one or more type of logs you would like to collect. Complete the remaining details required to finish the cluster creation and select Create.

    Screenshot showing how to select log type.

    Note

    If no option is selected, then only AKS service logs will be available.

  5. Log Analytics is enabled when the cluster is successfully created. All monitoring capabilities can be accessed under your cluster's Monitoring section.

    Screenshot showing Monitoring section for cluster in the Azure portal.

Enable Log Analytics using portal after cluster creation

  1. In the Azure portal top search bar, type "HDInsight on AKS clusters" and select Azure HDInsight on AKS clusters from the drop-down list. On the HDInsight on AKS cluster pools page, select your cluster name from the list page.

    Screenshot showing search option for getting started with HDInsight on AKS Cluster.

    Screenshot showing selecting the HDInsight on AKS Cluster you require from the list.

  2. Navigate to the "Monitor settings" blade, select Enable Log Analytics. Choose one or more type of logs you would like to collect, and click Save.

    Screenshot showing how to select more log types.

    Note

    If no option is selected, then only AKS service logs will be available.

Access the log tables and run queries using the portal

  1. From the Azure portal, select your cluster pool or cluster of choice to open it.

  2. Navigate to the Monitoring section and select the Logs blade to query and analyze the collected data.

    Screenshot showing logs in the Azure portal.

  3. A list of commonly used query templates is provided to choose from to simplify the process or you can write your own query using the provided console.

    Screenshot showing queries in the Azure portal.

    Screenshot showing New queries in the Azure portal.