The past twelve years have been quite a ride, first in the support organization, and then in engineering - working on OpsMgr first, then the prototype of what later became Application Insights, and in the last couple of years in the Advisor 'reset' - which then became Azure Operational Insights and more recently core of Operations Management Suite.

The team has done an amazing job so, and I am sure they will continue towards the vision of modernizing management. I am also sure they will continue to be listen to feedback and be customer driven. Not every team at Microsoft gets kudo's from "the Register" for showing customer interest, after all.

But the time has come for me to focus on different priorities in my life. 

You can still find me at my personal blog, and on Flickr/Twitter/Facebook/LinkedIn.

Shanti.

What do we mean by Wire Data?

The solution uses a custom module for both SCOM and Direct Agent, to hook into the Windows Network Stack and poll statistics from it. The data is then uploaded to OMS (directly from each agent, even in SCOM attach mode).

In OMS Log Search you can then filter and group data to view information about the top agents and top protocols. Or you can look into when certain computers (IP addresses/MAC addresses) communicated with each other, for how long, and how much data was sent--basically, you view metadata about network traffic, all powered by search – like most other solutions.

The data that the solution produces is pretty self explanatory, if you have been using OMS Log Search and are a little bit familiar with TCP/IP and network communications. Our main goal was that this process should be lightweight – we don’t go extremely deep in the network stack to conserve performance. With default settings we err on the side of dropping something – if necessary - rather than completeness / full fidelity of the data at the expense of performance. This said, the Wire Data module defines certain configuration properties (described later in this post) that can be tweaked.

Type=WireData and Type=SecurityEvent : a ‘better together’ use case

Here I want to illustrate a less obvious use case.

Let’s therefore not spend time on the canned drill down, that currently presents a breakdown of agents, protocols, etc – those should be hopefully self explanatory - and let’s start with one of the ‘Common Queries’ provided, a bit less in view:

Type=WireData | measure Sum(TotalBytes) by ProcessName

this lets me have a peek at an interesting dimension of this data: we often are able to tell which PROCESS the network communication was sent to/from:

(in the screenshot I added a ‘Where’ as I had too many…)

So here I notice that some traffic was sent from a suspiciously-named process ‘DancingPigs.exe’.

If I click it in the results above I get to the next query, which shows me that traffic: 

I can see those are a bunch of ‘outbound’ communications over various protocols (HTTPS, SMB, etc). Sounds like a user has executed something he shouldn’t have clicked on?

Since I also have the ‘Security and Audit’ solution in this workspace, I can lookup (thru a SUB-SEARCH – read my previous post here if you haven’t) those Security Events which have the SAME ProcessName field value (the field on both types has values in the same format) as those Wire Data logs:

Type=SecurityEvent ProcessName IN {Type:WireData "DancingPigs.exe" | distinct ProcessName}

and once I got this information, I can have a look at the Accounts that have started that process, according – again – to the Security Logs:

Type=SecurityEvent ProcessName IN {Type:WireData "DancingPigs.exe" | distinct ProcessName} | measure count() by Account

Hope the above gave a small demonstration of some interesting use case for this type of data.

How is data collected? Can I tweak collection?

As wrote also earlier, the solution uses a custom module for both SCOM and Direct Agent, to hook into the Windows Network Stack and poll statistics from it.

Our main goal was that this process should be lightweight and not affect the agent – therefore we don’t go extremely deep in the network stack to be gentle on machine’s performance. With default settings we err on the side of dropping something – if necessary - rather than having full completeness or fidelity at the expense of performance. This said, the Wire Data module defines the following configuration properties:

Which means, if you have SCOM, can be tweaked thru Overrides (at your own risk – do your own performance measurement to evaluate the impact of tweaking these limits, especially on busy servers with a lot of traffic.

On behalf of the team, we hope you will enjoy this solution and find it useful. It’s just a first step, ‘getting our toes wet’ into this space of network traffic visibility in OMS.

Please send us feedback and feature requests (or bug reports) as usual on the UserVoice feedback forum.

Now, before I continue, if you have just stumbled here but have no previous experience with the search syntax, notice that this is the ninth installment of a Series that walks thru the concepts of Operations Management Suite (formerly Microsoft Azure Operational Insights) Search Syntax – while the full documentation and syntax reference is here, these posts are meant to guide your first steps with practical examples. I’ll start very simple, and build upon each example, so you can get an understanding of practical use cases for how to use the syntax to extract the insights you need from the data.

In my first post I introduced filtering, querying by keyword or by a field’s exact value match, and some Boolean operators.

In the second post I built upon the concepts of the first one, and introduced some more complex flavors of filters that are possible. Now you should know all you need to extract the data set you need.

In the third post I introduced the use of the pipeline symbol “|” and how to shape your results with search commands.

In the fourth post I introduced our most powerful command – measure �� and used it with just the simplest of the statistical functions: count().

In the fifth post I expanded on the measure command and showed the Max() statistical function.

In the sixth post I continued with measure’s statistical functions – I showed how Avg() is useful with Performance data among other things.

In the seventh post I continued with another of measure’s statistical functions – Sum() – and introduced the where command.

In the eight post I introduced the sub-searches feature that is accessed with the IN operator.

Today I want to introduce you to DISTINCT. As the name hopefully suggests, this command provides the list of distinct values for a field. Sure, the same could be achieved with measure count() command as well:

Type=Event | Measure count() by Computer

But if all you are interested in is the just LIST of distinct values and not the count of documents that have that value, then DISTINCT makes for a cleaner/more readable output and shorter syntax

Type=Event | Distinct Computer

This is particularly useful in sub-searches – where all you want is really a distinct list of ‘something’ to pass in as filter value for an outer search. [Note – if you are confused about this last statement, read the previous post https://blogs.msdn.com/b/dmuscett/archive/2015/05/30/operations-management-suite-log-search-how-to-part-viii-the-in-operator-and-subsearches.aspx ]

Happy searching!

In my first post I introduced filtering, querying by keyword or by a field’s exact value match, and some Boolean operators.

In the second post I built upon the concepts of the first one, and introduced some more complex flavors of filters that are possible. Now you should know all you need to extract the data set you need.

In the third post I introduced the use of the pipeline symbol “|” and how to shape your results with search commands.

In the fourth post I introduced our most powerful command – measure – and used it with just the simplest of the statistical functions: count().

In the fifth post I expanded on the measure command and showed the Max() statistical function.

In the sixth post I continued with measure’s statistical functions – I showed how Avg() is useful with Performance data among other things.

In the seventh post I continued with another of measure’s statistical functions – Sum() – and introduced the where command.

I am very excited to be writing about an important addition to our search syntax: we recently enabled the ‘IN’ operator (along with ‘NOT IN’), which allows you to use subsearches: a subsearch is a search that includes another search as an argument. They are contained in curly brackets within another "primary" or "outer" search. The result of a subsearch (often a list of distinct results) is then used as an argument in its primary search.

You can use subsearches to match subsets of your data that you cannot describe directly in a search expression, but which can be generated from a search. For example, if you're interested in using one search to find all events from "computers missing security updates" you need to design a subsearch that first identifies that "computers missing security updates" before it finds events belonging to those hosts.

So, we could express ‘computers currently missing security updates” as follows:

Type:RequiredUpdate UpdateClassification:"Security Updates" TimeGenerated>NOW-25HOURS | measure count() by Computer

Once we have this list, we can use this search as inner search to feed the list of computers into an outer (primary) search that will look for events for those computers. We do this by enclosing the inner search in curly brackets and feeding its results as possible values for a filter/field in the outer search using the ‘IN’ operator. This query would look like:

Type=Event Computer IN {Type:RequiredUpdate UpdateClassification:"Security Updates" TimeGenerated>NOW-25HOURS | measure count() by Computer}

that’s it!

Also notice the time filter I used in the ‘inner’ search: since I am counting on ‘System Update Assessment’ Solution to have a snapshot of all machines every 24 hours, I am making that inner query more lightweight and precise by only making that go back in time a day; the outer search instead still respects the time selection in the user interface, pulling up events from the last 7 days. [Also check my previous blog post about time in search]

Another thing to notice is that, since we are really only using the results of the inner search as a filter value for the outer one, you can still apply commands in the outer search, i.e. we can still group the above events with another measure command:

Type=Event Computer IN {Type:RequiredUpdate UpdateClassification:"Security Updates" TimeGenerated>NOW-25HOURS | measure count() by Computer} | measure count() by Source

Generally speaking, you want your inner query to execute fast – we have some service-side timeouts for it – and to return a small amount of results. If the inner query returns more results, we will truncate the result list, which will cause also the outer search to potentially give incorrect results.

Another rule is that the inner search currently needs to provide ‘aggregated’ results – or in other words, it must contain a ‘measure’ command; you cannot currently feed ‘raw’ results into an outer search.

Also, there can be only ONE ‘IN’ operator (and it must be the last filter in the query), and multiple IN operators cannot be OR’d – this essentially prevents running multiple subsearches: the bottom line is that only one sub/inner search is possible for each ‘outer’ search.

Even with these limits, this addition enables all new kinds of correlated searches, and allows you to define something similar to ‘groups’ (of computers, or users, of files – whatever the fields in your data contain). Some more examples:

All updates missing from machines where Automatic Update setting is disabled

Type=RequiredUpdate Computer IN {Type=UpdateAgent AutomaticUpdateEnabled!=Enabled | Measure count() by Computer} | Measure count() by KBID

All error events from machines running SQL Server (=where SQL Assessment has run)

Type=Event EventLevelName=error Computer IN {Type=SQLAssessmentRecommendation | measure count() by Computer}

All security events from machines that are Domain Controllers (=where AD Assessment has run)

Type=SecurityEvent Computer IN { Type=ADAssessmentRecommendation | measure count() by Computer }

Which other accounts have logged on to the same computers where account BACONLAND\jochan has logged on?

Type=SecurityEvent EventID=4624   Account!="BACONLAND\\jochan" Computer IN { Type=SecurityEvent EventID=4624   Account="BACONLAND\\jochan" | measure count() by Computer } | measure count() by Account

I am sure you will come up with other interesting search scenarios now.

Happy searching!

This post aims at showing you how you can use Operational Insights Search to see which machines/environments are sending data, get an understanding of what data (and how much of it) is being received by the service in given amounts of time. This should help with both identifying problematic/missing machines (machines whose data is never seen) and derive an understanding of how the amount of data collected has an impact on usage and billing metrics of the service.

The first place to look at is the ‘Servers and Usage’ tile (one of the ‘small’ tiles in the Overview screen):

This tile shows the amount of data metered during the current metering cycle (which is midnight-to-midnight UTC). If you are already 4 hours into the current day and the count is still on zero, the tile colors red, like in the screenshot above.

Once you drill into the tile in the respective page – you get additional information about your usage and the amount of data that was metered in the last 7 days, a calculated daily average for those 7 days, and a breakdown of data by Intelligence Pack. Some intelligence packs collect a lot more data than other ones, and some are configurable and can be tuned up or down in this sense (i.e. Log Management). In Operations Manager, you can even create some of your own rules being counted as part of those IP’s, with a more granular collection policy.

Note that usage/metering information is tracked as data as received by the service, accumulated, and updated periodically. The information about ‘usage’ on this page can be slightly stale (1 or 2 hours delay). Information is presented in megabytes/gigabytes here, and broken down by the sending ‘intelligence pack’.

[Edited May 2015 – the list of management groups and the link to access the list of direct agents, described below, have moved to the ‘Settings’ tile, under ‘Connected Sources’]

The other blades in this page show you directly connected agents and Operations Manager management groups that have been registered with Operational Insights.

The list of management groups represents those that are currently registered with the service. You need to register an Operations Manager management group thru the Operations Manager console before it will show up here.

The amount of agents and the ‘data freshness’ status is not actually tracked on our end at this stage, but it is only inferred by the UI based on the presence of data. This is different from the old ‘servers’ page in Advisor (in the Configuration Assessment scenario and screens) that was maintaining a stateful list of all SCOM agents. We don’t actually maintain such a list in the new system – agent management and enrollment is better accomplished from the Operations Manager environment, as machines might naturally come and go there is no need to synchronize them ‘both ways’ as a list in the cloud. We use instead data in Search to see whether data is stamped as belonging to a certain ‘Computer’, and how many and which ones of those we see, to provide counts. Similarly, data freshness uses search: we look at the most recent piece of data for a given management group and report how old that is. We can’t guarantee data for the recent time is complete, just that there is some recent data. More data from the recent period might still being indexed and will show up in the system later.

In the case of management groups, if the last piece of data is older than 2 weeks, we offer a link to ‘remove’ the management groups: this permanently deletes the registration – should that management group ever come back online, its data will be rejected, and you would have to register it again from the Operations Manager console.

Note – for ‘directly connected servers’, we would like to provide an agent management experience that allows to list the agents that have been registered, and eventually remove (de-register) them like you can do with management groups once they are stale. Track this item for that feature https://feedback.azure.com/forums/267889-azure-operational-insights/suggestions/6949226-implemented-feature-to-remove-managed-management

Clicking on either the ‘servers connected directly’ or one of the tiles representing specific Operations Manager management groups takes you to search.

The drill down queries are similar in each case, let’s look at directly connected servers first: Direct agents will be seen as if coming from a fake management group “00000000-0000-0000-0000-000000000001", with this search:

MG:"00000000-0000-0000-0000-000000000001"  | Measure Max(TimeGenerated) as LastData by Computer | Sort Computer

The default query uses the Measure command to show the latest indexed piece of data grouped by the field ‘Computer’. Adjusting the time selector to an older period of time might show you more machines that used to report data in the past but might have dropped off. If you never see a machine listed that you expect to see, that’s an indication you need to troubleshoot it.

Note – some log types might not populate the ‘Computer’ field – see requirements for IIS Logs https://blogs.technet.com/b/momteam/archive/2014/09/19/iis-log-format-requirements-in-system-center-advisor.aspx

By drilling down into one of the computers, you will see all of its data, and you can then get a breakdown by any of the dimensions in the data with the measure count() command. A first easy assessment to understand what volume of data is to get a count of record grouped by each ‘Type’ (or even more simply, peek at the ‘Type’ facet/filter on the left side under ‘narrow your results’):

In this screenshot you can see what data my laptop had sent in the first few hours – Security Events and other Windows Events account for the majority of the data, followed by a few configuration changes and just 1 record of each type for those assessment that run infrequently. Exporting this to Excel I can quickly get a percentage and understand what’s the relative ‘weight’ of each Type.

Similar to the ‘servers connected directly’ drill-down, each Operations Manager management group has a similar drill down where the actual (real, in this case) management group ID is used to filter the data.

If you don’t want to look at directly connected agents and Operations Manager agents separately, you can run the same Measure command against a star ‘*’ filter – which means ‘all data’:

*  | Measure Max(TimeGenerated) as LastData by Computer | Sort Computer

or get the same breakdown by type for the whole environment, to better understand my overall ‘usage’

* | Measure count() by Type

this query is, btw, among the seeded ‘Saved Searches’ in the Portal. It’s under the ‘general exploration’ category, and named ‘Distribution of data types’:

Another interesting field you need to know about is ‘SourceSystem’: since Operational Insights can index data send by agents/management groups or ingest directly from Windows Azure Diagnostics (WAD) storage (typically for PaaS or IaaS Azure role instances), this field tracks where the data comes from. You’ll see there are a couple of varieties of ‘Operations Manager’ (‘OpsManager’, ‘OpsMgr’ – depending on intelligence pack/workflow) that essentially mean this data is ‘agent based’ (either directly connected agent or Operations Manager – but the data comes from the agent-style channel). On the contrary, some data for Log Management is tagged with SourceSystem=AzureStorage, to indicate that data is coming from WAD.

*  | Measure Max(TimeGenerated) as LastData by SourceSystem

and besides freshness, you can, again, also see quantities with measure count() command

*  | Measure count() by SourceSystem

Here you see in my environment, most data is agent-based and only a little is coming from WAD. YMMV.

Filtering by SourceSystem=AzureStorage, you can use some of the same techniques above to check what type of data is coming from one type of connection as opposed to the other one.

SourceSystem=AzureStorage | measure count() by Computer

SourceSystem=AzureStorage | measure count() by Type

Last note, records coming from WAD (SourceSystem=AzureStorage) also feature some additional Azure-specific properties, when compared to their agent-based equivalents, i.e. try grouping based on Deployment Id:

SourceSystem=AzureStorage | measure count() by AzureDeploymentID

Often a few more properties (depending on type) are also present referring to Role, RoleInstance and other Azure concepts.

I hope this was useful to understand how to slice and dice your data to understand usage patterns, machines that are not reporting, discover those that are sending the most data, and in general to get a better understanding of what data is flowing in the system.

Happy Searching!

For example, this is my simple recipe for finding SQL Backup Failures

Type=Event EventID=3041 EventLog=Application Source=MSSQLSERVER

That’s it. I can see in my workspace I had a few of those back in January:

I can slightly massage the output to get better readability by selecting only certain fields piping my results into the SELECT command

Type=Event EventID=3041 EventLog=Application Source=MSSQLSERVER | Select Computer,RenderedDescription

Now if you want to keep monitoring this, you can save it in your favorite searches and pin it to your Dashboard, so you can take a look, even on the go with the Mobile app!

In my case I am investigating an occurrence in the past, so I want to understand why the failure occurred, so the next logical step would be to look for the time when the backup failure occurred, by interacting with the time bars on the top left under ‘narrow your results’ – clicking on the bars and narrowing down till we see the few individual events and a narrow enough time period

and now I want to see WHAT ELSE was happening on the same machine, so I add a filter for the ‘Computer’ field, and I remove everything else from my search:

Computer="BaconSCOM.BaconLand.com"

This shows me WHAT ELSE I know about that machine around that time frame, and I see, among other things that are always found (i.e. events, securityevents don’t come at regular intervals, but all the times), I see there is ONE configuration change – interesting

Let’s drill into that… and we see that someone had installed a Defragmentation tool on the server around the same time as the backup failures!

I think we now understand why the backup failed that day…Operational Insights made it super easy to get to the root cause!

In my first post I introduced filtering, querying by keyword or by a field’s exact value match, and some Boolean operators.

In the second post I built upon the concepts of the first one, and introduced some more complex flavors of filters that are possible. Now you should know all you need to extract the data set you need.

In the third post I introduced the use of the pipeline symbol “|” and how to shape your results with search commands.

In the fourth post I introduced our most powerful command – measure – and used it with just the simplest of the statistical functions: count().

In the fifth post I expanded on the measure command and showed the Max() statistical function.

in the sixth post I continued with measure’s statistical functions – I showed how Avg() is useful with Performance data among other things.

By now you should have grasped how ‘measure’ works, therefore I will not actually spend time on the Sum() function, other than this mention – if you want to see an example of its use, refer to this blog post https://blogs.msdn.com/b/dmuscett/archive/2014/09/20/w3c-iis-logs-search-in-system-center-advisor-limited-preview.aspx where I already showed how to use it to get aggregate amount of traffic download to a given IP from a webserver (IIS logs):

Type=W3CIISLog | Measure Sum(scBytes) by cIP

Interesting/additional thing to notice before we move on: you can use Max() and Min() with numbers, datetimes and strings. with strings, they basically get sorted alphabetically and you get first and last.

You cannot however use Sum() – which does a REAL calculation – with anything but numerical fields. Same applies to Avg().

Where

The last command we have is Where.

Where works like a filter, but it can be applied in the pipeline to further filter ‘aggregated’ results that have been produced by a Measure command – as opposed to ‘raw’ results that get filtered at the beginning of a query.

Given this query:

Type=PerfHourly  CounterName="% Processor Time"  InstanceName="_Total" | Measure Avg(SampleValue) as AVGCPU by Computer

I can add another pipe “|” character and the Where command to only get computers whose average CPU is above 80%

Type=PerfHourly  CounterName="% Processor Time"  InstanceName="_Total" | Measure Avg(SampleValue) as AVGCPU by Computer | Where AVGCPU>80

You see what this represents?

If you are a system center person, talking in ‘management pack terms’, if this was a rule, the first part would be the ‘data source’, while the where command becomes the condition detection. (see my other dissertation on ‘rules’ and ‘searches’ similarities in this other post https://blogs.msdn.com/b/dmuscett/archive/2014/11/05/iis-mp-event-alerting-rules-s-opinsights-searches-equivalents.aspx )

Where’s the write action, you might ask?

Well, if you bind this to a tile in ‘My Dashboard’, this essentially is a ‘monitor’ that lets you see in real time if machines are under CPU pressure – soon even on your phone – as shown below, in the bottom two tiles, both as a list and as a number: you basically always want the number to be zero and the list to be empty – otherwise it indicates an alert condition, essentially, and you can peek which machines are under pressure:

Welcome to monitoring in the modern world!

For other style of ‘write actions’, consider this idea in our feedback forum for the actual creation of alerts and other notification (email etc) whenever those ‘where’ criteria (really if you think of them, those are ‘thresholds’) are met https://feedback.azure.com/forums/267889-azure-operational-insights/suggestions/6519198-run-saved-search-on-a-schedule-raise-alert-and-or or this other one to kick off an automation runbook in response to a query matchin results https://feedback.azure.com/forums/267889-azure-operational-insights/suggestions/6658045-add-ability-to-execute-runbook-automation-from-aoi

This concludes the planned ‘HowTo’ series on the search syntax. I hope it was useful and enjoyable. The idea was to let new users read one post a day and having learned the syntax and be productive with it in a week!

As we add more capabilities and commands I will continue it. Please remember the official syntax reference can be found here https://technet.microsoft.com/en-us/library/dn500940.aspx and that Stefan Roth has also produced a handy cheat sheet on his blog https://stefanroth.net/2014/11/05/microsoft-azure-operational-insights-search-data-explorer-cheat-sheet/

I will also likely blog some more scenario-focused posts, i.e. same use case and how to go thru more examples of ‘searches in action’.

I also keep updating this other post as I come up with some new useful queries https://blogs.msdn.com/b/dmuscett/archive/2014/10/19/advisor-searches-collection.aspx

Remember the feedback forum is always open for you to vote and ask what you would like us to improve in the service at https://feedback.azure.com/forums/267889-azure-operational-insights/

Happy searching!

Why did I do this, you might be asking? Well, you must have realized that, with Azure Operational Insights’s Log Management and Search capabilities, you would be able to collect the same event logs (mostly System and Application event logs is what is used by IIS) and using the Search query syntax, get those results – wait, if those were alerting rules, this means when one of those events was seen, it would generate al alert, right?

Well, you must have noticed that in a dashboard we feature a couple of style of tiles, in particular you can see a time-based distribution of occurrences of search results (those events) – so you can keep under control if and WHEN those errors happen, in a easy to reach way:

or – if you don’t want those events to ever occur (after all, if they occur, you would alert, right?) you can keep them under control with the numeric tile and set a threshold – if more than ZERO (or more, if you want to tolerate a few – and you can add a time filter to the query to make it act on the most recent time period… this would be your ‘repeated event’ criteria, in ‘management packs’ terms…) then you can color the tile:

In the future, we’d like to enable long running or scheduled searches that would produce ‘actions’ – i.e. produce an alert, notify via email, kick off a runbook…. and visualize your dashboard on your smartphone! Check out and vote those ideas, and let’s enable monitoring in a modern hybrid world!

And now, here’s the searches I produced by running this small experiment – Try them out, and have fun searching! (and you can try repeating the experiment yourself with other MP’s too )

A script has not responded within the configured timeout period
EventID=2216 Source="Microsoft-Windows-IIS-W3SVC-WP" EventLog=Application

A server side include file has included itself or the maximum depth of server side includes has been exceeded
EventID=2221 Source="Microsoft-Windows-IIS-W3SVC-WP" EventLog=Application

ASP application error occurred
EventID=500 OR EventID=499 OR EventID=23 OR EventID=22 OR EventID=21 OR EventID=20 OR EventID=19 OR EventID=18 OR EventID=17 OR EventID=16 OR EventID=9 OR EventID=8 OR EventID=7 OR EventID=6 OR EventID=5 Source="Active Server Pages" EventLog=Application

HTTP control channel for the WWW Service did not open
EventID=1037 Source="Microsoft-Windows-IIS-W3SVC" EventLog=System

HTTP Server could not create a client connection object for user
EventID=2208 Source="Microsoft-Windows-IIS-W3SVC-WP" EventLog=Application

HTTP Server could not create the main connection socket
EventID=2206 Source="Microsoft-Windows-IIS-W3SVC-WP" EventLog=Application

HTTP Server could not initialize its security
EventID=2201 Source="Microsoft-Windows-IIS-W3SVC-WP" EventLog=Application

HTTP Server could not initialize the socket library
EventID=2203 Source="Microsoft-Windows-IIS-W3SVC-WP" EventLog=Application

HTTP Server was unable to initialize due to a shortage of available memory
EventID=2204 Source="Microsoft-Windows-IIS-W3SVC-WP" EventLog=Application

ISAPI application error detected
EventID=2274 OR EventID=2268 OR EventID=2220 OR EventID=2219 OR EventID=2214 Source="Microsoft-Windows-IIS-W3SVC-WP" EventLog=Application

Module has an invalid precondition
EventID=2296 Source="Microsoft-Windows-IIS-W3SVC-WP" EventLog=Application

Module registration error detected (failed to find RegisterModule entrypoint)
EventID=2295 Source="Microsoft-Windows-IIS-W3SVC-WP" EventLog=Application

Module registration error detected (module returned an error during registration)
EventID=2293 Source="Microsoft-Windows-IIS-W3SVC-WP" EventLog=Application

Only one type of logging can be enabled at a time
EventID=1133 Source="Microsoft-Windows-IIS-W3SVC" EventLog=System

SF_NOTIFY_READ_RAW_DATA filter notification is not supported in IIS 8
EventID=2261 Source="Microsoft-Windows-IIS-W3SVC-WP" EventLog=Application

The configuration manager for WAS did not initialize
EventID=5036 Source="Microsoft-Windows-WAS" EventLog=System

The directory specified for caching compressed content is invalid
EventID=2264 Source="Microsoft-Windows-IIS-W3SVC-WP" EventLog=Application

The Global Modules list is empty
EventID=2298 Source="Microsoft-Windows-IIS-W3SVC-WP" EventLog=Application

The HTTP server encountered an error processing the server side include file
EventID=2218 Source="Microsoft-Windows-IIS-W3SVC-WP" EventLog=Application

The server failed to close client connections to URLs during shutdown
EventID=2258 Source="Microsoft-Windows-IIS-W3SVC-WP" EventLog=Application

The server was unable to acquire a license for a SSL connection
EventID=2227 Source="Microsoft-Windows-IIS-W3SVC-WP" EventLog=Application

The server was unable to allocate a buffer to read a file
EventID=2233 Source="Microsoft-Windows-IIS-W3SVC-WP" EventLog=Application

The server was unable to read a file
EventID=2226 OR EventID=2230 OR EventID=2231 OR EventID=2232 Source="Microsoft-Windows-IIS-W3SVC-WP" EventLog=Application

WAS detected invalid configuration data
EventID=5174 OR EventID=5179 OR EventID=5180 Source="Microsoft-Windows-WAS" EventLog=System

WAS encountered a failure requesting IIS configuration store change notifications
EventID=5063 Source="Microsoft-Windows-WAS" EventLog=System

WAS encountered an error attempting to configure centralized logging
EventID=5066 Source="Microsoft-Windows-WAS" EventLog=System

WAS encountered an error attempting to look up the built in IIS_IUSRS group
EventID=5153 Source="Microsoft-Windows-WAS" EventLog=System

WAS encountered an error trying to read configuration
EventID=5172 OR EventID=5173 Source="Microsoft-Windows-WAS" EventLog=System

WAS is stopping because it encountered an error
EventID=5005 Source="Microsoft-Windows-WAS" EventLog=System

WAS received a change notification, but was unable to process it correctly
EventID=5053 Source="Microsoft-Windows-WAS" EventLog=System

WAS terminated unexpectedly and the system was not configured to restart it
EventID=5030 Source="Microsoft-Windows-WAS" EventLog=System

Worker process failed to initialize communication with the W3SVC service and therefore could not be started
EventID=2281 Source="Microsoft-Windows-IIS-W3SVC-WP" EventLog=Application

WWW Service did not initialize the HTTP driver and was unable to start
EventID=1173 Source="Microsoft-Windows-IIS-W3SVC" EventLog=System

WWW service failed to configure the centralized W3C logging properties
EventID=1135 OR EventID=1134 Source="Microsoft-Windows-IIS-W3SVC" EventLog=System

WWW Service failed to configure the HTTP.SYS control channel property
EventID=1020 Source="Microsoft-Windows-IIS-W3SVC" EventLog=System

WWW service failed to configure the logging properties for the HTTP control channel
EventID=1062 Source="Microsoft-Windows-IIS-W3SVC" EventLog=System

WWW Service failed to copy a change notification for processing
EventID=1126 Source="Microsoft-Windows-IIS-W3SVC" EventLog=System

WWW Service failed to enable end point sharing for the HTTP control channel
EventID=1175    Microsoft-Windows-IIS-W3SVC    System

WWW service failed to enable global bandwidth throttling
EventID=1071 OR EventID=1073 Source="Microsoft-Windows-IIS-W3SVC" EventLog=System

WWW service property failed range validation
EventID=5067 Source="Microsoft-Windows-WAS" EventLog=System

In my first post I introduced filtering, querying by keyword or by a field’s exact value match, and some Boolean operators.

In the second post I built upon the concepts of the first one, and introduced some more complex flavors of filters that are possible. Now you should know all you need to extract the data set you need.

In the third post I introduced the use of the pipeline symbol “|” and how to shape your results with search commands.

In the fourth post I introduced our most powerful command – measure – and used it with just the simplest of the statistical functions: count().

In the fifth post I expanded on the measure command and showed the Max() statistical function. And gave ‘homework’ to try Min() on your own

This time, I will show you one other favorite of mine to be used with measure, the Avg() statistical function. As you can imagine, this allows you to calculate the average value for some field, and group results (as usual with Measure – we discussed this in the 4th post) by some (other or same) field.

This is useful in a variety of cases, my favorite one being Performance data, but there are other interesting use cases.

Let’s start with Performance data, anyhow, as it probably is the easiest to understand.

Before I do that, let me remind you that – at the time of this writing – we only collect specific fabric-related performance counters for VMM and HyperV hosts as part of the ‘Capacity’ Intelligence Pack.

We are anyhow tracking ideas to collect Custom-defined Performance Counters on our feedback forum – go ahead and vote on them if you would like us to enable that functionality!

• Collect Custom Performance Counters from Windows Agents https://feedback.azure.com/forums/267889-azure-operational-insights/suggestions/6519061-collect-custom-windows-performance-counters
• Collect Custom Performance Metrics from Unix/Linux Agents https://feedback.azure.com/forums/267889-azure-operational-insights/suggestions/6519187-collect-performance-metrics-for-unix-and-linux-ser
• Collect Custom Performance Counters for Azure PaaS Cloud Services from Windows Azure Diagnostics Storage https://feedback.azure.com/forums/267889-azure-operational-insights/suggestions/6519356-collect-custom-performance-counters-from-windows-a

For these ideas, we would also like to understand the granularity of collection you expect to see – the current performance data collected by Capacity Intelligence Pack is only indexed in pre-computed hourly aggregations. What does this mean? It means that for each Computer\Performance Object\Performance Counter Name\Instance Name (as in perfmon) we currently index ONE RECORD in search for each hourly interval (24/day).

Let’s see what this means. If I search for ALL performance data, the most basic query would be

Type=PerfHourly

the first thing you notice is that OpInsights shows you also charts of those performance counters

this is convenient, of course, but let’s scroll to the bottom and look at the actual records that are behind those charts, and how do they look like

Look at the screenshot above and the two sets of fields I circled in red and notice a few things:

• the first set lets you identify the Windows Performance Counter Name, Object Name and Instance Name in your query filter. These are the fields you probably will most commonly use as facets/filters.
• ‘SampleValue’ is the actual value of the counter (more in a second on this…)
• Type=PerfHourly = this is an hourly aggregate
• TimeGenerated: it is at 21 o’clock. It is the aggregation for that hourly bucket from 20:00 to 21:00
• SampleCount - the aggregation was computed using 12 samples (one every 5 minutes)
• the minimum, maximum and 95th percentile for the hourly bucket was – in this case for memory for one of my VM’s – always 6144 (megabytes)
• ‘SampleValue’ – back to it - since this record is a hourly aggregate, ‘SampleValue’ is populated with the AVERAGE for the hourly bucket.
• this field is what is used to plot the performance charts
• we left this name (rather than a more explicit Avg) because this way we can later introduce ‘raw’ (non pre-aggregated) performance data (where ‘SampleValue’ will be the actual value picked from perfmon at a specific time, unprocessed) and by keeping this field name the same, both the per chart as well as the most common queries you’d write will continue to work, unchanged, across ‘raw’ or ‘aggregated’ data.

So after all this explaining of the PerfHourly record ‘shape’, and having read the previous blog posts, you can now understand how to use measure Avg() to aggregate this type of very ‘numerical’ data.

Simple example:

Type=PerfHourly  ObjectName:Processor  InstanceName:_Total  CounterName:"% Processor Time" | Measure Avg(SampleValue) by Computer

By now, with all the previous blog posts and examples and filtering and using measure with other functions, this should be clear – we select the CPU Total Time performance counter and we ask an Average by Computer. Easy.

You might notice that, since ‘SampleValue’ is already an average, you are really asking an average of an average. That’s correct with Type=PerfHourly at the moment, but you see where this will be more precise and useful when we’ll have a raw Type=Perf, or similar, not pre-aggregated. For the time being, we suggest always throwing a filter on TimeGenerated (see 2nd post where we talked about time filters) to restrict the operation to a small/recent dataset – i.e. the last 4 hours, not 7 days!

So our query above becomes

Type=PerfHourly  ObjectName:Processor  InstanceName:_Total  CounterName:"% Processor Time" TimeGenerated>NOW-4HOURS | Measure Avg(SampleValue) by Computer

Try it now. You will see this recent average will generally be higher.

Or with a twist, you could calculate the average of the Maximum hourly values, i.e.

Type=PerfHourly  ObjectName:Processor  InstanceName:_Total  CounterName:"% Processor Time" TimeGenerated>NOW-4HOURS | Measure Avg(Max) by Computer

Even more interesting – one of my absolute favorite scenarios – is aggregating/correlating data ACROSS machines, something that was VERY HARD to do with System Center Operations Manager and something I really wanted to simplify.

Let’s imagine we have a set of hosts in some sort of farm where each node is equal to any other one and they just do all the same type of work and load should be roughly balanced… I could get their counters all in one go with the following query and get averages for the entire farm! Let’s start with choosing the computers

Type=PerfHourly AND (Computer=”SERVER1.contoso.com” OR Computer=”SERVER2.contoso.com” OR Computer=”SERVER3.contoso.com”)

(note that today you have to do an OR – but this could become a subsearch in the future, that could be used to express a ‘dynamic group’ – check out and vote this idea on our feedback forum in this regard https://feedback.azure.com/forums/267889-azure-operational-insights/suggestions/6519209-allow-subqueries-in-the-search-language-in-not 

In any case, now that we have the computers, we also only want to select two KPI – % CPU Usage and % Free Disk Space. This part of the query becomes:

Type=PerfHourly  InstanceName:_Total  ((ObjectName:Processor AND CounterName:"% Processor Time") OR (ObjectName="LogicalDisk" AND CounterName="% Free Space")) AND TimeGenerated>NOW-4HOURS

If we now put all together – machines and counters:

Type=PerfHourly  InstanceName:_Total  ((ObjectName:Processor AND CounterName:"% Processor Time") OR (ObjectName="LogicalDisk" AND CounterName="% Free Space")) AND TimeGenerated>NOW-4HOURS AND (Computer=”SERVER1.contoso.com” OR Computer=”SERVER2.contoso.com” OR Computer=”SERVER3.contoso.com”)

and now that we have this very specific selection, the measure Avg() command can tell us the average not by computer, by across the farm, simply by grouping by CounterName:

Type=PerfHourly  InstanceName:_Total  ((ObjectName:Processor AND CounterName:"% Processor Time") OR (ObjectName="LogicalDisk" AND CounterName="% Free Space")) AND TimeGenerated>NOW-4HOURS AND (Computer=”SERVER1.contoso.com” OR Computer=”SERVER2.contoso.com” OR Computer=”SERVER3.contoso.com”) | Measure Avg(SampleValue) by CounterName

which gives me this beautiful compact view over a couple of my farm’s KPI’s – at a glance I know:

and this, btw, looks great in a dashboard:

I am hoping with these examples you are starting to see how easy it is to quickly look at your environment’s data with the use of the search filters and commands.

Till next time, happy searching!

In my first post I introduced filtering, querying by keyword or by a field’s exact value match, and some Boolean operators.

In the second post I built upon the concepts of the first one, and introduced some more complex flavors of filters that are possible. Now you should know all you need to extract the data set you need.

In the third post I introduced the use of the pipeline symbol “|” and how to shape your results with search commands.

In the fourth post I introduced our most powerful command – measure – and used it with just the simplest of the statistical functions: count().

So let’s continue from where we left, and let’s explore some of the other statistical functions you can use with measure.

Measure Max() and Measure Min()

There are various scenarios where these are useful. I will only illustrate Max() and will leave Min() as an exercise for the reader – since it does the exact opposite of the other one.

Let’s start with a simple example. If I query for ‘Advisor’ Configuration Assessment Alerts, they have a ‘Severity’ property which is either 0,1 or 2 (meaning info/warning/critical):

If I want to see what is the HIGHEST value for all of the alerts given a common ‘Computer’ (the ‘group by’ field), I can write

Type=Alert | Measure Max(Severity) by Computer

and it will show me that – for the computers that had ‘Alert’ records, most of them have at least a critical one, and the ‘BaconSCOM’ machine has a warning as ‘worst’ severity:

This of course works well with NUMBERs, but it can also work with DateTime fields, i.e. it is very useful to check what is the last/most recent timestamp for any piece of data indexed for each computer, i.e.

When was the most recent configuration change reported by change tracking Intelligence Pack for each machine?

Type=ConfigurationChange | Measure Max(TimeGenerated) by Computer

Hope this got you started on measure Max() – now try measure Min().

In the next Installment we’ll look at the AVG (average) statistical function with the measure command – particularly interesting with performance/capacity data!

Stay tuned for that, and happy searching!