Sovereignty choices for monitoring Azure workloads

  • Article

Sovereignty requirements often apply not only to the application and infrastructure services that are used as part of a cloud workload, but also to the management solutions that are used in the operation and administration of that workload.

Organizations that have to meet strict sovereignty requirements should identify monitoring solutions that satisfy both operational and compliance requirements. This ensures that design patterns for workload monitoring are used by the teams that are planning workload migrations.

In this article, you'll learn about the different goals and best practices for monitoring, and compare the cloud-native and bring your own solution approaches.

Understand goals for logging and monitoring

Understanding the behavior of resources that are deployed in the cloud is critical to deliver a dependable solution. Although monitoring is often included as a component of cloud workloads, it's important to understand that monitoring is often implemented for different reasons and for the benefit of different stakeholders.

If an organization is going to design a holistic monitoring solution in the cloud, it would be useful to highlight the different objectives that organizations often encounter.

Monitor for performance

Monitoring the performance of a workload can take many forms, including monitoring the health of an application service, the availability of solution components, and the speed and responsiveness of the solution. This type of monitoring is performed in near-real time to identify system issues as soon as possible and avoid downtime.

Metrics from this type of monitoring can also be collected and aggregated to analyze performance trends. This type of monitoring and the data it produces is often used by application and infrastructure teams that manage resources, as well as operations and support teams that respond to events and incidents.

Monitor for security

Monitoring is often implemented to provide an organization with detective controls that can help manage risk. Monitoring security events can help an organization respond quickly and minimize the effect of threats. Threat monitoring can look for patterns that correspond with known attack techniques, and maintaining event data over time can allow an organization to conduct forensic investigations and perform root cause analysis.

Data collected from security monitoring is often used by security teams, including operations analysts and threat hunters, as well as IT operations, assurance, and audit teams.

Monitor for service management

Along with performance and security monitoring that looks at the behavior of a workload, organizations may implement extra monitoring to look at the state of the workload. This type of monitoring is often used to verify that IT Service Management objectives are being met. Service Management domains such as configuration management, change control, and software version currency often require monitoring the version or configuration of a resource to validate deployment in a known good state.

This kind of monitoring is often used by IT operations teams, application and infrastructure teams, and security teams to identify unauthorized changes.

Use best practices for monitoring and diagnostics

As organizations plan their monitoring solutions, it's useful to review some best practices for implementing cloud-native monitoring for solutions that are deployed in Azure. The following articles contain recommendations for designing cloud-based monitoring solutions:

Cloud-native monitoring vs. bring your own solution

Many organizations already have mature monitoring solutions in-place to monitor on-premises systems, and a common choice when planning a cloud migration is whether to adopt a cloud-native monitoring solution or adapt an existing solution for use in the cloud.

Both of these approaches have their benefits and drawbacks, so we recommend that organizations evaluate both approaches to ensure a good alignment with their operational and sovereignty requirements.

Use logging and monitoring as a service

Azure offers a selection of cloud-native services that organizations can use to create a holistic monitoring solution:

  • Azure Monitor can centrally collect and analyze log data from cloud workloads.
  • Log Analytics provides a graphical interface for creating and running queries against collected log data.
  • Azure Monitor Insights provides monitoring experiences in the Azure portal curated by Microsoft.
  • Application Insights works with Azure Monitor to provide application performance management features for customer-written code.
  • Microsoft Sentinel can be used with Azure Monitor for security orchestration, automation, and response (SOAR).
  • Microsoft Defender for Cloud is a cloud-native application protection platform (CNAPP) that works with Azure Monitor to protect cloud-based applications from threats.

While an organization can choose to develop their monitoring approach from scratch using Azure Monitor and Log Analytics, many organizations can benefit from the curated experiences in services like Azure Monitor Insights and Microsoft Defender for Cloud.

These services might not provide the same level of granularity when it comes to selecting locations for data residency, so organizations should understand where and how their data is stored if they choose to incorporate non-regional services into their monitoring strategy.

Extend on-premises monitoring solutions to Azure

There are several ways that organizations can continue to take advantage of their on-premises monitoring solutions for applications with highly sensitive data that can't be monitored using PaaS monitoring solutions.

  • For IaaS workloads, agent-based monitoring solutions can continue to be included in virtual machine images.
  • Application Performance Monitoring solutions can continue to be compiled with customer-developed code.
  • Logging servers can be deployed in Azure using virtual machines to minimize client traffic across WAN links.
  • Logs can be sent to storage accounts, streamed with Event Hubs, or accessed via API.

All of these approaches can help organizations transition their operations model to the cloud while maintaining a higher level of operational sovereignty for their on-premises monitoring systems. However, these approaches may also add extra costs as legacy monitoring solutions consume cloud resources like virtual machines and cloud storage.

Another approach that can help organizations to transition their operations to the cloud is to stream monitoring data from Azure Monitor to on-premises solutions that are provided by Azure Monitor Partners.

Select monitoring solutions for Azure workloads

The following scenarios highlight some of the monitoring solutions that organizations can use to monitor workloads, including workloads with strict sovereignty requirements:

Monitor Azure resources using regional and non-regional services

  • Data Sources and Instrumentation: Collect platform and activity logs natively using Azure Monitor. Collect logs from IaaS resources using Azure Monitor agents. Collect runtime telemetry using Application Insights.
  • Collection and Storage: Aggregate log data for an individual workload in an Azure Monitor Workspace. Aggregate log data across the enterprise in Azure Data Lake by streaming logs using Event Hubs.
  • Analysis and Diagnosis: Analyze logs using Log Analytics or Azure Data Explorer. Generate insights using PaaS solutions such as Monitor Insights and Defender for Cloud. Create automation and orchestration using Microsoft Sentinel.

Monitor Azure resources using regional services only

  • Data Sources and Instrumentation: Collect platform and activity logs using Azure Monitor. Collect runtime telemetry using Application Insights.
  • Collection and Storage: Aggregate log data for an individual workload in an Azure Monitor Workspace deployed in your desired region. Stream log data with Event Hubs to a data lake in your desired subscription.
  • Analysis and Diagnosis: Analyze logs using Log Analytics or Azure Data Explorer.

Monitor Azure resources using on-premises solutions

  • Data Sources and Instrumentation: Capture logs with Azure Monitor and export to on-premises solution using storage account, event hubs, or API. Capture logs directly using third party agents.
  • Collection and Storage: Aggregate and archive log data on-premises.
  • Analysis and Diagnosis: Use existing on-premises solutions for analysis and diagnosis.