Monitor Nexus Kubernetes cluster

Each Nexus Kubernetes cluster consists of multiple layers:

• Virtual Machines (VMs)
• Kubernetes layer
• Application pods

Figure: Sample Nexus Kubernetes cluster

On an instance, Nexus Kubernetes clusters are delivered with an optional Container Insights observability solution. Container Insights captures the logs and metrics from Nexus Kubernetes clusters and workloads. It's solely your discretion whether to enable this tooling or deploy your own telemetry stack.

The Nexus Kubernetes cluster with Azure monitoring tool looks like:

Figure: Nexus Kubernetes cluster with Monitoring Tools

Extension onboarding with CLI using managed identity auth

Documentation for starting with Azure CLI, how to install it across multiple operating systems, and how to install CLI extensions.

Install latest version of the necessary CLI extensions.

Monitor Nexus Kubernetes cluster

This how-to guide provides steps to enable monitoring agents for the collection of System logs from these VMs using Azure Monitoring Agent. The instructions further capture details on how to set up log data collection into a Log Analytics workspace.

Prerequisites

• Cluster administrator access to the Nexus Kubernetes cluster.

• To use Azure Arc-enabled servers, register the following Azure resource providers in your subscription:

  • Microsoft.HybridCompute
  • Microsoft.GuestConfiguration
  • Microsoft.HybridConnectivity

Register these resource providers, if not done previously:

az account set --subscription "{the Subscription Name}"
az provider register --namespace 'Microsoft.HybridCompute'
az provider register --namespace 'Microsoft.GuestConfiguration'
az provider register --namespace 'Microsoft.HybridConnectivity'

• Assign an Azure service principal to the following Azure built-in roles, as needed. Assign the service principal to the Azure resource group that has the machines to be connected:

Role	Needed to
Azure Connected Machine Resource Administrator  or Contributor	Connect Arc-enabled Nexus Kubernetes cluster VM server in the resource group and install the Azure Monitoring Agent (AMA)
Monitoring Contributor or Contributor	Create a Data Collection Rule (DCR) in the resource group and associate Arc-enabled servers to it
User Access Administrator, and Resource Policy Contributor or Contributor	Needed if you want to use Azure policy assignment(s) to ensure that a DCR is associated with Arc-enabled machines
Kubernetes Extension Contributor	Needed to deploy the K8s extension for Container Insights

Monitor Nexus Kubernetes cluster

Prerequisites

There are certain prerequisites the operator should ensure to configure the monitoring tools on Nexus Kubernetes Clusters.

Container Insights stores its data in a Log Analytics workspace. Log data flows into the workspace whose Resource ID you provided during the installation of the Container Insights extension. Else, data funnels into a default workspace in the Resource group associated with your subscription (based on Azure location).

An example for East US may look like follows:

• Log Analytics workspace Name: DefaultWorkspace-<GUID>-EUS
• Resource group name: DefaultResourceGroup-EUS

Run the following command to get a pre-existing Log Analytics workspace Resource ID:

az login

az account set --subscription "<Subscription Name or ID the Log Analytics workspace is in>"

az monitor log-analytics workspace show --workspace-name "<Log Analytics workspace Name>" \
  --resource-group "<Log Analytics workspace Resource Group>" \
  -o tsv --query id

To deploy Container Insights and view data in the applicable Log Analytics workspace requires certain role assignments in your account. For example, the "Contributor" role assignment. See the instructions for assigning required roles:

• Log Analytics Contributor role: necessary permissions to enable container monitoring on a CNF (provisioned) cluster.
• Log Analytics Reader role: non-members of the Log Analytics Contributor role, receive permissions to view data in the Log Analytics workspace once you enable container monitoring.

Install the cluster extension

Sign-in into the Azure Cloud Shell to access the cluster:

az login

az account set --subscription "<Subscription Name or ID the Provisioned Cluster is in>"

Now, deploy Container Insights extension on a provisioned Nexus Kubernetes cluster using either of the next two commands:

With customer pre-created Log analytics workspace

az k8s-extension create --name azuremonitor-containers \
  --cluster-name "<Nexus Kubernetes cluster Name>" \
  --resource-group "<Nexus Kubernetes cluster Resource Group>" \
  --cluster-type connectedClusters \
  --extension-type Microsoft.AzureMonitor.Containers \
  --release-train preview \
  --configuration-settings logAnalyticsWorkspaceResourceID="<Log Analytics workspace Resource ID>" \
  amalogsagent.useAADAuth=true

Use the default Log analytics workspace

az k8s-extension create --name azuremonitor-containers \
  --cluster-name "<Nexus Kubernetes cluster Name>" \
  --resource-group "<Nexus Kubernetes cluster Resource Group>" \
  --cluster-type connectedClusters \
  --extension-type Microsoft.AzureMonitor.Containers \
  --release-train preview \
  --configuration-settings amalogsagent.useAADAuth=true

Validate Cluster extension

Validate the successful deployment of monitoring agents’ enablement on Nexus Kubernetes Clusters using the following command:

az k8s-extension show --name azuremonitor-containers \
  --cluster-name "<Nexus Kubernetes cluster Name>" \
  --resource-group "<Nexus Kubernetes cluster Resource Group>" \
  --cluster-type connectedClusters

Look for a Provisioning State of "Succeeded" for the extension. The "k8s-extension create" command may have also returned the status.

Customize logs & metrics collection

Container Insights provides end-users functionality to fine-tune the collection of logs and metrics from Nexus Kubernetes Clusters. See the instructions for Configure Container insights agent data collection for more information.

Note

Container Insights does not collect logs from the kube-system namespace by default. To collect logs from the kube-system namespace, you must configure the agent to collect logs from the kube-system namespace. This can be done by removing the kube-system namespace from the excludedNamespaces field in the ConfigMap following the configMap configuraiton approach.

[log_collection_settings]
  [log_collection_settings.stdout]
    # In the absense of this configmap, default value for enabled is true
    enabled = true
    # exclude_namespaces setting holds good only if enabled is set to true
    # kube-system,gatekeeper-system log collection are disabled by default in the absence of 'log_collection_settings.stdout' setting. If you want to enable kube-system,gatekeeper-system, remove them from the following setting.
    # If you want to continue to disable kube-system,gatekeeper-system log collection keep the namespaces in the following setting and add any other namespace you want to disable log collection to the array.
    # In the absense of this configmap, default value for exclude_namespaces = ["kube-system","gatekeeper-system"]
    exclude_namespaces = ["gatekeeper-system"]
    # If you want to collect logs from only selective pods inside system namespaces add them to the following setting. Provide namepace:controllerName of the system pod. NOTE: this setting is only for pods in system namespaces
    # Valid values for system namespaces are: kube-system, azure-arc, gatekeeper-system, kube-public, kube-node-lease, calico-system. The system namespace used should not be present in exclude_namespaces
    # collect_system_pod_logs = ["kube-system:coredns"]

  [log_collection_settings.stderr]
    # Default value for enabled is true
    enabled = true
    # exclude_namespaces setting holds good only if enabled is set to true
    # kube-system,gatekeeper-system log collection are disabled by default in the absence of 'log_collection_settings.stderr' setting. If you want to enable kube-system,gatekeeper-system, remove them from the following setting.
    # If you want to continue to disable kube-system,gatekeeper-system log collection keep the namespaces in the following setting and add any other namespace you want to disable log collection to the array.
    # In the absense of this configmap, default value for exclude_namespaces = ["kube-system","gatekeeper-system"]
    exclude_namespaces = ["gatekeeper-system"]
    # If you want to collect logs from only selective pods inside system namespaces add them to the following setting. Provide namepace:controllerName of the system pod. NOTE: this setting is only for pods in system namespaces
    # Valid values for system namespaces are: kube-system, azure-arc, gatekeeper-system, kube-public, kube-node-lease, calico-system. The system namespace used should not be present in exclude_namespaces
    # collect_system_pod_logs = ["kube-system:coredns"]

Extra resources

• Review workbooks documentation and then you may use Operator Nexus telemetry sample Operator Nexus workbooks.
• Review Azure Monitor Alerts, how to create Azure Monitor Alert rules, and use sample Operator Nexus Alert templates.