Azure Operator Nexus observability

The Operator Nexus observability framework provides operational insights into your on-premises instances. The framework supports logging, monitoring, and alerting (LMA), analytics, and visualization of operational (platform and workloads) data and metrics.

Figure: Operator Nexus Logging, Monitoring and Alerting (LMA) Framework

The key highlights of Operator Nexus observability framework are:

• Centralized data collection: Operator Nexus observability solution is based on a collection of all the data in a central place. In this place, you can observe the monitoring data from all of your on-premises instances.
• Well-defined and tested tooling: The solution relies on Azure Monitor that collects, analyzes, and acts on telemetry data from your cloud and on-premises instances.
• Easy to learn and use: The solution makes it easy for you to analyze and debug problems with the ability to search the data from within or across all of your cloud and on-premises instances.
• Visualization tools: You create customized dashboards and workbooks per your needs.
• Integrated Alert tooling: You create alerts based on custom thresholds. You can create and reuse alert templates across all of your instances.

This article helps you understand Operator Nexus observability framework that consists of a stack of components:

• Azure Monitor collects and aggregates logging data from the Operator Nexus components
• Azure Log Analytics Workspace (LAW) collects and aggregates logging data from multiple Azure subscriptions and tenants
• Analysis, visualization, and alerting are performed on the aggregated log data.

Platform Monitoring

Operator Nexus gives you visibility into the performance of your deployments that consist of infrastructure resources. You need the logs and metrics to be collected and analyzed from these platform resources. You gain valuable insights from the centralized collection and aggregation of data from all sources, compared with from dis-aggregated data.

These logs and metrics are used to observe the state of the platform. You can see the performance and analyze what's wrong. You can analyze what caused the situation. Visualization helps you configure the required alerts and under what conditions. For example, you can configure the alerts to be generated when resources are behaving abnormally, or when thresholds have been reached. You can use the collected logs and analytics to debug any problems in the environment.

Monitoring Data

Operator Nexus observability allows you to collect the same kind of data as other Azure resources. The data collected from each of your instances can be viewed in your LAW.

You can learn about monitoring Azure resources here.

Collection and Routing

Operator Nexus observability allows you to collect data for each infrastructure resource. The set of infrastructure components includes:

• Network fabric that includes CEs, TORs, NPBs, management switches, and the terminal server.
• Compute that includes Bare Metal Servers.
• Undercloud Control Plane (Kubernetes cluster responsible for deployment and managing lifecycle of overall Platform).

Collection of log data from these layers is enabled by default during the creation of your Operator Nexus instance. These collected logs are routed to your Azure Monitor LAW.

You can also collect data from the tenant layers created for running Containerized and Virtualized Network Functions. The log data that can be collected includes:

• Collection of syslog from Virtual Machines (used for either VNFs or CNF workloads).
• Collection of logs from Kubernetes clusters and the applications deployed on top.

You'll need to enable the collection of the logs from the tenant Kubernetes clusters and Virtual Machines. You should follow the steps to deploy the Azure monitoring agents. The data would be collected in your Azure LAW.

Operator Nexus Logs storage

Data in Azure Monitor Logs is stored in tables where each table has its own set of unique properties.

All resource logs in Azure Monitor have the same fields followed by service-specific fields; see the common schema.

The logs from Operator Nexus platform are stored in the following tables:

Table	Description
Syslog	Syslog events on Linux computers using the Log Analytics agent
ContainerInventory	Details and current state of each container.
ContainerLog	Log lines collected from stdout and stderr streams for containers
ContainerNodeInventory	Details of nodes that serve as container hosts.
InsightMetrics	Metrics collected from Server, K8s, Containers.
KubeEvents	Kubernetes events and their properties.
KubeMonAgentEvents	Events logged by Azure Monitor Kubernetes agent for errors and warnings.
KubeNodeInventory	Details for nodes that are part of Kubernetes cluster
KubePodInventory	Kubernetes pods and their properties
KubePVInventory	Kubernetes persistent volumes and their properties.
KubeServices	Kubernetes services and their properties
Heartbeat	Records logged by Log Analytics agents once per minute to report on agent health

Operator nexus metrics

The 'InsightMetrics' table in the Logs section contains the metrics collected from Bare Metal Machines and the undercloud Kubernetes cluster. In addition, a few selected metrics collected from the undercloud can be observed by opening the Metrics tab from the Azure Monitor menu.

Figure: Azure Monitor Metrics Selection

See Analyze metrics with Azure Monitor metrics explorer for details on using this tool.

Workbooks

Workbooks combine text, log queries, metrics, and parameters for data analysis and the creation of multiple kinds of rich visualizations. You can use the sample Azure Resource Manager workbook templates for Operator Nexus Logging and Monitoring to deploy Azure Workbooks within your Azure LAW.

Alerts

You can use the sample Azure Resource Manager alarm templates for Operator Nexus alerting rules. You should specify thresholds and conditions for the alerts. You can then deploy these alert templates on your on-premises environment.

Log Analytic Workspace

A Log Analytics Workspace (LAW) is a unique environment to log data from Azure Monitor and other Azure services. Each workspace has its own data repository and configuration but may combine data from multiple services. Each workspace consists of multiple data tables.

A single LAW can be created to collect all relevant data or multiple workspaces based on operator requirements.