Monitoring: Best Practices 

General design principles

Understand the top goals of monitoring

Understand the different components of monitoring

Provide enough information to start the troubleshooting process

Provide multiple channels for notification

Provide a context for the monitoring information

Design toward specific monitoring scenarios

Optimize your monitoring feature for different organizational sizes

Increase the signal-to-noise ratio

Provide explanations

Allow users to capture and save monitoring data

Use the appropriate graph for the type of information you are displaying

Use a standard alert scale across your application

Teams developing monitoring UI should use the following principles to create monitoring UI that will enhance performance, increase security, and allow the administrator to react proactively to problems on servers and on the network.

General design principles

The following principles will help ensure that your monitoring output is easy to read and well presented.

• IT administrators appreciate being able to view lots of information in a small area in their monitoring displays.
• Provide a high-level “dashboard” as a jumping-off point into more detail.
• Ensure that navigation around the monitoring tool and access to troubleshooting tools is efficient.
• To increase efficiency, allow easy customization of the monitoring UI. Users should be able to select what they want to monitor at any given time and the format they want to view the data in. They should be able to select the placement of their tools.
• UI text should be appropriate to the IT user: consistent, clear, concise, helpful, and professional in tone.

Understand the top goals of monitoring

When you create UI for monitoring, the three most important user-experience goals are:

• Ensure that the UI allows the IT administrator to easily troubleshoot and determine the health of system components such as a service, application, or server role.
  For more information, see:
  Understand the different components of monitoring
  Provide enough information to start the troubleshooting process
• Ensure that users can be notified of problems via e-mail and other methods. Provide simple daily or weekly reports that summarize key characteristics of the system.
  For more information, see:
  Reporting: Best Practices
  Provide multiple channels for notification
• Ensure that the UI conveys to the administrator which system components are important to monitor, what the “recommended” values for those components are, and what appropriate corrective actions are.
  For more information, see:
  Provide a context for the monitoring information

Understand the different components of monitoring

The basic elements that make up monitoring are events, thresholds, notifications, and health.

Events

Events are triggered on a system when a programmatically set condition occurs—for example, when a service starts or fails to start. In Microsoft Windows Server 2003, most events are logged in Event Viewer. Events are used by Microsoft Operations Manager (MOM) for monitoring analysis.

Thresholds

Threshold refers to the point on a scale that must be reached in order to trigger a response to an event. The response can be an alert, a notification, or a script being run. Either an application or a user can set a threshold.

Notifications

Notification refers to how an IT administrator is informed that something (an event or a response) has occurred. Notifications can take many forms, including:

• Alerts in an application, such as MOM, Performance Monitor, and other monitoring applications
• E-mail messages
• Instant Messenger messages
• Netsend dialog boxes that appear on an IT administrator’s screen
• Pager text messages
• Taskbar pop-up notifications

Health

Health refers to the set of metrics that define the state of the functionality being monitored, including functionality defined by the user. For example, the overall health of a file server could be defined by the performance of read/write disk access, CPU usage, network performance, and disk fragmentation. You should define the initial model and values for health for your component. Administrators can then customize the components of their health model and the values that represent a “healthy” state for each of their components.

Provide enough information to start the troubleshooting process

It is not enough to tell IT administrators that there is a problem with their system. Enough information must be given so that the IT administrator can begin to work to fix the problem.

• Provide both probable cause and possible solutions.
• Ensure that information which helps the user take action is near the monitoring information.
• Ensure that the correct tools for common problems are easily accessible at the point of notification and that they are applicable to the probable cause or solution.
• Allow users to update, document, and share the information used to start troubleshooting.
• Log between 24 and 48 hours of data so that when a problem arises, users will have data available to troubleshoot the problem. Some applications need longer periods of data and some need shorter periods. Match the log length to your scenarios.

Provide multiple channels for notification

Typically, IT administrators like to be notified of problems by the following methods, in order of preference:

• E-mail messages
• Pager text messages
• Instant Messenger messages
• Pop-up notifications from the taskbar

Provide a context for the monitoring information

Often a value from the system is only meaningful when it is compared to a normal range or other data. Provide normal or baseline data to make the monitoring information most valuable.

Baselines and suggested normal ranges

The system can provide these ranges or you can allow users to calculate them. These ranges allow users to compare their data with “normal” or “recommended” data. Users should be able to determine at a glance when a value is out of range and is cause for concern.

Trending (time)

Provide trending information that allows users to see how the values of variables change over time. This generally requires more than 48 hours of data to be logged.

Mean (average)

Mean is calculated by adding all values and dividing by the number of values. This allows users to see the “typical” value. Allow users to calculate averages for different periods. Display clearly how the average is calculated and the data set to which the mean applies.

Highs and lows

Show the range of the data over a certain period of time. Ensure that the period of time is displayed.

Median

The median is the point at which half the values are higher and half the values are lower. This helps show distribution of values. Provide an explanation of what a median is.

Scope

State clearly what is being monitored and what is not. For example, if the monitoring tool simply pings a server to see if it is running, it should be made clear that the tool is not monitoring whether clients can get to that server.

Design toward specific monitoring scenarios

By providing information targeted toward specific and common questions that you think users will want answered, you ensure that the data you are presenting will be optimized and useful.

• Take the time to understand what your customers need to monitor. Remember to optimize the UI for the most common scenarios and not for the rare cases.
• Target your default monitoring toward the identified core scenarios. This will help reduce “noise.”
• Use your expertise in your feature to identify the top variables to monitor, and provide these as default values. Prioritize these in the UI.
• Provide preconfigured monitoring views that match real-world situations.
• Provide useful views that match how users think about looking for information. For example, allow users to access monitoring data by starting at a tool, a specific computer, or through server roles.

Optimize your monitoring component for different organizational sizes

Different types of organizations have different monitoring needs. Target your monitoring data toward these organizational sizes to ensure that users get the information they need.

Small Organization (SORG)

The primary focus of SORG IT administrators is troubleshooting problems after they have been identified. If monitoring in itself is in fact an administrator activity, it is not a primary activity. Monitoring generally is undertaken as part of the troubleshooting process.

• SORG IT administrators are reactive, so monitoring should be proactive. Monitoring should intelligently identify problems and notify the users of the problems via e-mail and other means.
• Because of infrequent monitoring, SORG IT administrators require much more explanation about what is being monitored and what the results mean.
• Work to reduce the need for monitoring. A good feature is one that does not break or need to be monitored. If you feel something needs to be monitored, determine whether you can make the feature more robust so that it does not need monitoring. Ask yourself, for example, if the feature can have a degree of “self-healing.”
• Keep the monitoring UI simple. Small organizations will not be doing sophisticated monitoring; your adding lots of monitoring features will only make monitoring itself difficult.

Medium Organization (MORG)

MORG IT administrators are a little more proactive than SORG IT administrators but are still very reactive. MORG IT administrators have limited time to monitor and just want a “red light/green light” indication of health. Adding a “yellow light” status will help IT administrators become more proactive.

• MORG IT administrators monitor to capture the big and obvious problems and to make sure the system is functioning. Monitoring to fine-tune the system is not a main scenario.
• Provide explanations with the monitoring information to help get the troubleshooting process started. Consider automating common tasks that IT administrators must perform upon receiving negative monitoring information (for example, ping, trace route, etc.).
• Provide an overview of the system that users can occasionally glance at.
• Provide preconfigured views. MORG IT administrators do not have the knowledge or time to determine what their most important monitoring views are and to set them up.
• Keep the monitoring UI simple. These organizations will not be doing sophisticated monitoring; lots of monitoring features will only make monitoring itself more difficult.

Large Organization (LORG) /Enterprise

IT administrators in these types of organizations require more specialized views. Monitoring at this level is granular, in that detailed and specific information is needed.

• Allow LORG IT administrators to easily customize their monitoring views.
• Users in LORGs and enterprises are often dedicated exclusively to monitoring, and as a result, appreciate dense data. Sacrifice ease of learning for long-term ease of use.
• Provide mechanisms for administrators to collaborate. They will want to send information to each other for further analysis.
• LORGs and enterprises use monitoring to fine-tune the system.

Increase the signal–to-noise ratio

IT administrators are often overloaded with information that is of no value to them. This is “noise” that can get in the way of the important information. For example, displaying a lot of informational messages makes it difficult for administrators to spot the information that is of most value (for example, critical error messages).

Increase the signal in the following ways:

• Use a hierarchy.
• Place important information at the top of the hierarchy.
• Hierarchy can be achieved through:
  • Position – placing important items prominently
  • Ordering – putting more important information before less important information
  • Highlighting – using some visual indication, such as text or an icon, to indicate the important information
• Use preconfigured monitoring views.
• Allow users to customize the information and indicate what they think is important, so that it can be elevated in the UI.
• Group similar events.
• Consider having a global overview of the entire system visible at all times.

Reduce the noise in the following ways:

• Don’t show the same event over and over again as the same failure reoccurs, or at least group the same events into one.
• Use filters and views.
• Tag events correctly. Only truly critical events should be tagged as critical.
• Allow users to tag events as ones they do not want to view. Allow users to re-tag events.
• Allow users to set thresholds that match their own monitoring scenarios. This allows them to define what is important in their own environment and reduces false alarms.
• Allow users to have granular control of what they are monitoring, such as sampling rate, how the data is viewed or displayed, and what is monitored.
• Provide the ability to zoom in and see more detailed data. Provide the ability to zoom out and see aggregated data. Allow users to hide unimportant events but still have them accessible.
• Avoid remote monitoring unless the sample period is sufficiently large to compensate for possible network latency issues and the occasional random aberrant value.
• Consider implementing some degree of root-cause analysis. Most problems occur due to some basic reason, such as network outage. In such cases, if 100 things fail in the system, instead of showing 100 things in the UI, show one entry for network outage. This requires some level of instrumentation and logic built into the application itself, but it is something worth considering for your application.

Provide explanations

Do not assume that your users will understand all the terms and information you provide. By providing explanations, you ensure that the data you provide is understandable and interpretable.

• Provide roll-overs to display specific data points such as a chart spike.
• Display information about how the monitoring view is filtered. For example, some variables or events might not be shown, not because there is no problem, but because the variable has been hidden. Place the filter mechanism, explanation of the filter, and the data itself all close together so it is clear that they relate to each other.
• Give easy access to variables that were excluded in a view.
• State when the last sample of data was captured.
• Reference the data source.
• Table headings and variable names in column and row headings should be links that pop up explanations of the variables, abbreviations, and acronyms.
• Provide links beside tables, which pop up explanations of the entire table.

Allow users to capture and save monitoring data

Often users need to use their monitoring data for further analysis or for reports. Provide a system flexible enough that they can take the data away and use it to fulfill their own unique goals.

• Ensure that data representing normal performance can be captured so that it can later be used as a baseline for troubleshooting.
• Allow users to log the information they are monitoring. Users should be allowed to easily specify what they want recorded. For example, they should be able to specify the duration, variables, format, and location of the log.
• Ideally, all logs from different areas of monitoring should be stored in a centralized place where one UI can be used to analyze the data. Based on your user scenarios, consider identifying which teams to partner with, so log data can be viewed as a coherent whole.
• Allow users to export their data for reports and further analysis (in formats such as .xls, .html, .txt, .csv). This is especially helpful in large organizations where there is escalation and collaboration in troubleshooting.

Use the appropriate graph for the type of information you are displaying

Different graph types provide different types of information even though the data may be the same. Make sure you know why you are using a particular type of graph.

Line graphs

• Line graphs are good for displaying one or more variables along a dimension such as time.
• Ranges, medians, and means can all be displayed simultaneously.

Tabular

• Table format is good for when users need to see precise numbers.
• Organize the table so that the important information is readily apparent. For example, put important information first or highlight it.
• This format is also good for showing a lot of dense information at once to users who will view the table repeatedly. In this case, the cells in the table can contain rich information such as graphical elements associated with numbers, or symbols to indicate state. While harder to learn initially, these elements are more efficient in the long run.

Histograms (bar graphs)

• Bar graphs allow values at a single point in time to be compared easily.
• Ranges, medians, and means can all be displayed simultaneously.
• Avoidthe use of stacked bar graphs. It is difficult to compare bar segments’ heights when they don’t start at the same baseline. It is better to use a histogram and put the values in clusters along the same baseline or break them out into separate graphs.
• Consider providing a details pane for a graph, where clicking a graph shows displays details about the graph in the pane.
• When using a line graph, show as few variables as possible—five at the most. This makes the graph easer to read.
• Avoid trying to convey too many messages in one graph. For example, avoid a pie chart that shows percentage of overall profit as the angle of the pie piece, growth over the last year as distance from the center point, and who the group is under as color, and group revenue as intensity of color, and group expenses as pie-piece height, etc.
  Exception: Showing many variables in a chart can be done when there are interesting interactions amongst the variables that can not otherwise be shown—for example, a spike in the number of event messages occurring at the same time as a drop in a performance variable.
• Clearly distinguish each data and graph element (line, bar, pie piece, range, average, median, etc.) from each other, as well as from the background of the graph.
• When using a graph to show percentage data, always use a pie chart: other data can be shown in addition to percentage data, but should not be presented in lieu of percentage data. For example, you can show the raw numbers associated with the percentages.
• Never use a graph to display a single data point. There is no more information in such a graph than if the number alone is presented. The graph just provides visual noise.
• Avoid the use of 3D in your charts, it only adds visual noise.
• Allow users to easily flip between different views of the same data.

Use a standard alert scale across your application

A standard alert scale makes all the data easy to compare and ensures that your output is accessible.

• Choose your colors with Localization in mind.
• Common colors for Western cultures are:
  • Red = Severe or critical
  • Yellow = Warning or informational
  • Green = Good
• The system primary default values for red, yellow, and green are
  • R = #FF0000
  • Y = #FFFF00
  • G = #00FF00
• Combine the colors with icons for users who are blind.
• Combine colors with words so that a screen reader can get status too.
• Ensure that each variable retains the same color across different types of graphs.
• Consider accessibility when choosing colors. Ensure that the colors can be configured for high-contrast mode.
• Ensure that information is not conveyed by color alone.
• Avoid too many custom status icons. Users will have to learn any new ones and there might be conflict with icons in other applications.