OMS is the new player in management space and its gaining momentum every day. In this blog post I want to demonstrate the capabilities around Log Search API using  Azure Automation and how it can help us in daily challenges.

Scenario : While managing our datacenters we face many challenges. When we face  a high utilization on a particular server or  slowness  in an application , we need to log into the server and collects/analyze many information to find the root cause.  Operations Manager is already tracking these and reporting  to us but without further  analysis  operator does not have any insight what's is causing the behavior on the target system. In our scenario we will use Azure Automation to  do the investigation using log search API  and send us the results.

General Requirements

First of all you will need an OMS subscription ( you can create one with free tier) and an Azure Automation account  . Free tiers will work perfect for testing. 

Runbook Requirements

Authentication

There are different ways to authenticate with Azure Resource Manager (ARM) . In this sample  I’ll be mainly using the ARM REST Api for  almost everything  so I will authenticate using  a Service Principal.  You can also  use native PowerShell cdmlets   to perform the same  . For details you can  check Tiander’s blog;

https://blogs.technet.microsoft.com/privatecloud/2016/04/05/using-the-oms-search-api-with-native-powershell-cmdlets/

To create a  Service principal  for unattended authentication to ARM ;

Go to  Active Directory tab  in Azure Portal

Select Applications

Add a new application suing the following option ;

Select Web Application  and give the application an URL.  this URL will be used only for identifying the application so any url will work. For this demo I will use https://myomsclient

On Active Directory Applications page  select  the application you created in last step.

Copy ClientID – This is the username  for our application

Generate a  new key  and select validity 1 or 2 years , this is the password for the service principal.

Copy both information as we will be  storing this as variables in Azure automation.

Role Assignment in Azure Resource Manager

After service principal is created  it needs to be given permissions to read the information about the  resources  in Azure Resource Manager.  To give Read right on all resources in your subscription  run the following command  in Azure Powershell.

login-AzureRMAccount

New-AzureRmRoleAssignment -RoleDefinitionName Reader –ServicePrincipalName <yourclientid>

To give permission to only OMS Workspace  ;

New-AzureRmRoleAssignment -RoleDefinitionName Reader -ServicePrincipalName <yourclientid>  -ResourceGroupName oi-default-east-us

oi-default-east-us  is the default name for OMS accounts created in East US. You can  run

Get-AzureRmResourceGroup to get the resource group name for your OMS resource group.

P.S. Replace ServicePrincipalID and SubscriptionID with your own values

Collecting Data

Now we  can authenticate to Azure Resource Manager and start querying OMS Log Search API.

For unattended authentication you will need to provide the service principal ID and access key as well as you AD tenant . Easiest way to provide this to the runbook is using Runbook assets . Runbook requires the following assets;

$SPAppID = Get-AutomationVariable -Name 'SPCertAppID'    # Variable Asset   - ClientID of the AD Application
$subscriptionid= Get-AutomationVariable -Name 'MsIntSubsID' # Variable Asset   -  Azure subscriptionID
$CLIENTKEY=Get-AutomationVariable -Name 'ArmclientKey'  # Variable Asset   -  Key to authenticate with the AS application
$resgrp=Get-AutomationVariable -Name 'OIEUSRG'     #Variable Asset   - OMS resource group   oi-default-east-us  in my case
$oiwssclab=Get-AutomationVariable -Name 'OIWS_SCLAB'     #Variable Asset   - OMS workspace Name
$hotmailcreds=Get-AutomationPSCredential -Name "myhotmail" # Credential Asset  : my Hotmail  credential
$mailto=Get-AutomationVariable -Name "Notificationrecp" #Variable Asset   -  Receipent email address
$tenant = Get-AutomationVariable -Name "ADTenant" #Variable Asset   - My Ad tenant name
$ARMUrl = "https://management.azure.com"
$ARMGraphUrl = "https://graph.windows.net"
$apiver=Get-AutomationVariable -Name 'ArmApiVersion'   # Variable asset    . I’m using 2015-03-20    as API Version

Complete runbook can be downloaded from ;

Performing Queries

Runbook utilizes search-oi function to perform the OMS search.  To make your own search  first define the query

$query="Type=SecurityEvent AccountType=user (EventID=4624)   AND TimeGenerated>NOW-8HOUR (Computer=""$server"") LogonTypeName=""2 - Interactive"" | top 5|select Account, Process,timegenerated"

and supply it as a parameter to search-oi function

     $interactivelogons=search-oi -query $query

Query result :

By using these technique you can modify/extend the searches performed in this runbook.

Email Findings

Runbook  will get all returned search queries and construct a html email body  .  We will send this mail to the recipient  using the Hotmail credentials . Here after using the code Hotmail will mpst probably will block it as suspicious activity. You need to go the  settings and allow this activity. If you use other email provider you might need to  enable some security so Azure Automation can authenticate and send email on your behalf.

After collecting the data I’ll send  the findings using the following code

$SMTPServer = "smtp.live.com"
$SMTPPort = "25"
$Username = "yourliveid@hotmail.com"
$Password = ""

$to = "<yourrecipientaddress>"

$subject = "OMS Search Demo"

$message = New-Object System.Net.Mail.MailMessage
$message.subject = $subject
$message.body = $body
$message.to.add($to)

$message.from = $username

$message.IsBodyHTML=$true

$message.AlternateViews
$smtp = New-Object System.Net.Mail.SmtpClient($SMTPServer, $SMTPPort);
$smtp.EnableSSL = $true
$smtp.Credentials = $hotmailcreds

$smtp.send($message)
write-output  "Mail Sent"

Webhook

Since we will be  triggering this runbook  from OMS/OpsMgr    easiest way to implement it  is to use Webhooks.

Webhooks are URLs which we can utilize to  trigger the runbook. All security is based on  the URL so when you create the webhook keep the URL safe.

Navigate to your automation Account / runbooks select  the runbook you have created and then select Webhooks

Create a new webhook

Give webhook a name,  enable it and set an end date. Here you need to copy  and save the URL  , after saving the webhook you wont be able to fetch it again!!!

From webhook run settings select Run on Azure  and save .

Now you will be able to trigger this runbook by Http POST.

Operations Manager Notification Channel

Now we will utilize this webhook  to sstart log search automation by using the following script

Param ($server,$alert,$data )

$uri = “https://s1events.azure-automation.net/webhooks?token=replacewithyourownWebhookURI”
$headers = @{"From"=Scom@contoso.local;"Date"="$((Get-date).ToString())"}

$vms  = @(
            @{ Server="$server";Alertname="$alert";Alertdata="$data";}
           
        )
$body = ConvertTo-Json -InputObject $vms

$response = Invoke-RestMethod -Method Post -Uri $uri -Headers $headers -Body $body

IN Operations Manager  you can trigger this  webhook using the  powershell script above in a command notification channel ;

"G:\MPauthoring\OMSNotification.ps1" -server '$Data[Default='Not Present']/Context/DataItem/ManagedEntityPath$\$Data[Default='Not Present']/Context/DataItem/ManagedEntityDisplayName$' -alert  '$Data[Default='Not Present']/Context/DataItem/AlertName$' -data '$Data[Default='Not Present']/Context/DataItem/AlertDescription$'

So all you need is to create a notification subscription  using this new channel  for the selected alerts and  Operations Manager will trigger  the runbook  Please keep in mind,  for  this scenario to work your Management server needs to be able to  access the webhook url.

Result

Here is the  email  generated after the automated search. I have queried almost all  the data I have for that specific server in this demo. In real life you can decide which details to fetch from OMS based on your own needs.

Happy automating and utilizing OMS !

You have probably heard about  Microsoft Operations Management Suite (OMS) . OMS  is the all-in-one hybrid cloud management solution for any enterprise to manage their workloads and applications on Azure or AWS, Windows Server or Linux, VMware or OpenStack .

For details pls check  Microsoft OMS

OMS is an all in one solution  which gives you the insights of your IT environment. It collects and analyses the data using  available solutions in the gallery.  You can also you very powerfully log search feature to  analyze the collected data and create dashboards.

Here is a sample landing page with all solutions enabled;

I’ll not go into the  details of OMS components in this blog post but rather focus on feeding additional data into OMS and visualizing it .

OMS utilizes either   the Microsoft Monitoring Agent directly or your Operations Manager environment to collect performance and event data from the servers.  Different solutions in OMS analysis collected  data and  gives you recommendations. 

Default data collection settings lets you to specify which events , performance data  and logs will be collected .

As you can see these not much to configure. These settings are generally enough  to capture the insights of most of the workloads we have  in an enterprise. When you have  the integration with Operations Manager it gets even better. With a little bit of MP authoring  you  can  start sending any information to OMS either as an event or a performance counter .

My Azure Iaas Dashboard

While playing around I decided to create a dedicated dashboard for my Azure consumption. MP is still under development but here is a preview dashboard ;

Captured performance data can also be visualized as a line graph;

I’ll share the MP tp capture these data on Part II of this blog. MP requires Azure subscriptions already discovered in Operations Manager  and  certificate to manage the subscription .  Details to follow…

Data Visualization Options

OMS has a nice  landing page and dashboard to visualize and analyze the data   but you are not limited to the web  page. Another advantage of OMS is the mobile apps . OMS has apps on every mobile platform so these dashboards and searched defined can be viewed in any platform anytime anywhere.  To get the app  visit https://www.microsoft.com/en-us/server-cloud/operations-management-suite/mobile-apps.aspx

In this  blog post I  will go thought the operating system deployment process with PXE boot step by step.  Since this scenario includes many technologies,  troubleshooting the  problems   takes some extra effort. I’ll try to show and examine all the technologies involved.

PXE Boot Step by Step

Process steps for the PXE boot is  as follows

• Client is booted from network 
• Client contacts DHCP for IP address
• Client receives IP from DHCP  and Boot Server, Boot File Name for the PXE boot 
• Client contact PXE  server and downloads the boot file
• Client  boots into Win PE
• Client looks for available task sequence
• and deployment starts…..

Based the  steps we  are facing the problem we need different tools/logs for troubleshooting

Before starting PXE

To test the PXE boot I have created a test task sequence and deployed it to All Unknown Computers collection with the following settings;

We first need to check references to see if anything is missing from the Distribution points.  Missing packages will cause task sequence to fail .

Step 1 - Enabling PXE Boot

In order to use PXE boot you need to enable this option from the BIOS of the computer. After enabling each vendor uses their own way of initiating the network boot. Most of the time its started either by pressing F12   or choosing network boot from the boot options.

Step 2 – Client Receives IP from DHCP Server

As first step of PXE  client needs an IP address from the DHCP server.    If the client does not getting the IP address best  option is to check network connectivity between client and the DHCP server. You can use a network trace to find out if your client is actually reaching the DHCP server itself.

Tools to use : Microsoft Network Monitor (MNM) or any other network package capture tool

To use  MNM   download the latest version and install it on the DHCP server. After the setup you need to  go to New Capture  and click Capture Settings

Here we are only interested in DHCP traffic so we will specify DHCP  as protocol in filter page and click apply. As

next step we will start the network capture and try to boot the client using PXE.

Client initiates the network boot. Here note the Mac address of the client as it will help us to track the request from the client.

Expected communication ;

First broadcast package shows up the mac address of the PXE client ;

If you don’t see this package this means broadcast messages are not  reaching the PXE server  so  you need to inspect the network devices in between your client and DHCP server.  Most probable cause  would be the switch cl’ent connected to don’t have the IPhelper setup .

After the broadcast received  CMPRi12 ( my DHCP server) replies and finally client gets the 192.168.1.5  ip address. If you inspect the reply message from DHCP  it also includes the Boot server and Boot file name ;

In order DHCP to work you need some configuration on the server side.  First DHCP server should be authorized and should have a scope, firewall should not block DHCP traffic.  Also we need to configure the following options for PXE boot

Option 66 Boot server : This is the PXE server   your ConfigMgr DP with PXE role

Option 67 Boot File Name : \smsboot\x64\wdsnbp.com

Option 60  PXEClient : PXEClient . You only need this if your PXE server and DHCP server is collocated. Since this is quite unlikely for a production environment  most of the time you wont need to configure this setting.

At this point we will switch to ConfigMgr DP  and look at the SMSPXE.log

Here I see my MAC address contacting  to the PXE server  but its not serviced. We got a warning message saying Architecture boot image  not found.

This message means PXE server does not have the required boot image available. You need to go to ConfigMgr console and Boot image properties. Under datasource enable  Deploy this boot image from the PXE enabled distribution point

After enabling this when PXE provide loads you will see your boot images (CAS00003.wim and CAS00004.wim in my environment) available in PXE

Step 3 – Downloading the boot image

Option 1 - machine is not in the ConfigMgr database and  boot  images are deployed to PXE .

Now we will retry the PXE boot  .  Here client contact PXE server and start downloading the boot image;

During this in  SMSPXE.log

Here we can again see  00:15:5D:00:6C:4F   our client  has contacted to PXE  server and  after checking we can see device is not in the database . We can also see that   advertisement CAS20001 will be used to deploy the client.

If you check the network trace of your PXE server you can see the TFTP traffic while downloading the boot image . You can capture this traffic by using  TFTP  as a capture filter.

Option 2 - If you try PXe boot  with a machine already registered in  ConfigMgr database  you will see that machine  will not be serviced by PXE.

Option 3 - If  the machine is serviced   before but  deployment  breaks before complementing  you will  see that machine will not be serviced by PXE.

As you can see machine is unknown but still will not be serviced. PXE has an internal protection mechanism to protect computers from being re-imaged   so when a computer is  serviced a record is being created in ConfigMgr database indicating the date and time it was serviced by PXE.

Here at this point if the deployment fails before the actual OS deployment  you will end up with  either

• a computer object named Unknown  and PXE  deployment will be cached on this resource
• If process breaks very early  you will not have any object created in ConfigMgr but PXE deployment will still be registered in database.

In both conditions that device will not be serviced by PXE component  until we clear this flag.  You can see the list of computers already serviced by  running the following query  in ConfigMgr database.

SELECT * from v_LastPXEDeployment

To clear this flag ;

• if you have can see the computer object named Unknown   with the details above  you can right click and run Clear the PXE Deployment
• If there is no such computer object  you need to import the computer again by using the details from that SQL table  and then you will be able to right click and run Clear the PXE Deployment

Step 4 – Boot into Win PE

After downloading the  boot image , client will boot into  Win PE and  will start the required advertisement.

If you receive any problems at this point SMSTS.log is your friend. You should enable command line support on the boot image and  open a cmd by pressing F8 then  check the SMSTS.log   for further troubleshooting.  

Good Luck …..

SQL Management Pack  offers extensive monitoring capabilities for monitoring SQL workloads.  You need to use the Operations Manager console to utilize the  full range of capabilities.

When I visit   some customers I realize that  DBA’s not always have access to the OpsMgr console and they receive alert notifications  mostly by email. If you check some  alerts you will find alert description lacking  sme important information. In order to get that  details you need to open alert context  from the OpsMgr console.

Example SQL Blocked Sessions Monitor

This  monitor is disabled by default and customers running  in house build applications with SQL backend tend to  monitor  blocked sessions closely. So  OpsMgr Admin enables this monitor for the SQL instances. an when a blocking session  occurs  we have an  alert in OpsMgr.

As  seen in screenshot we  can only see  that session 57 is blocked on  the default instance on a server. When SQL DBA receives that email  they need to go back  access the server to see   what was that session running    and which session is  blocking it etc .  In fact  query running on SQL  has all the details and its published under   Alert Context tab

Again this information is available only trough OpsMgr console  , Alert view.

Here SQL MP  runs a SQL query  compiles  and publishes all information about blocking sessions in property bags and hands then to the monitor. By altering the script slightly we can  publish more data and   change the alert description to show this information.

Here is the same alert with the modified monitor ;

As  seen in the alert details above now  blocked session details like program, login id, TSQL  and blocking session id   are all in the alert description.  So now just from the alert description DBA’s can see who is running  the blocked query  as well as the query and blocking session id.

Attached MP  has a new monitor called Blocked SPIDs targeted to SQL DB Engine.

SQL.BlockedSessions.xml.zip

For monitoring Lync we  have  a solid MP with strong capabilities.  We have capability to monitor core components  and by using Lync watcher nodes we can  simulate user  experience.   One key piece of information I found    missing in the MP is  the  number of users connected to front ends server at given  point in time. Most customers uses such counters to measure  how the service being consumed, which different endpoints being used etc.

Lync stores the user information in    RTCDYN  database  of Lync  in frontend servers.There are various scripts/queries to collect this information. Here I’ve used  the script sample from  Pat Richard ( https://www.ehloworld.com/269)  and  converted it to  feed data  to Operations Manager using  property bags .

After you start collecting this from Front End servers  then you can visualize  it using  Performance Widget, Object Detail Widget or in a performance report.

Update  ( 26/07/2015):

I have fixed the    wrong folder references and uploaded the MP again.

Microsoft.Lync.2013.Extensions.zip

One of the most common scenarios while monitoring the servers are monitoring  CPU and Memory.  We have default monitors in OS MP   which checks  the total processor utilization and Available MBs of memory.  These 2 are key  performance counters when we start troubleshooting any performance  related issue on servers.

On of the challenges about our monitors  is we get alerted when CPU or Memory under pressure  but in order to troubleshoot you need to know  which process is actually causing it . This key information is missing in our alerts.  Consider the situation where you  get those alerts when you are not at the office   , in this case you have no pointers from operations manager to diagnose the situation.

Diagnostic Tasks

In Operations Manager we  can attach a task to a monitor  to

• gather more information  - Diagnostic Tasks
• Take action to resolve the problem – Recovery Tasks

So all we need to attach a task to retrieve  top processes  when these monitors goes unhealthy.

Tracking High CPU Utilization

Here I have one server under load and both memory and processor monitors are in unhealthy state.

To find what is consuming  the CPU cycles we can run  List Top CPU Consuming processes  task from Windows Server Operating System state  view under Microsoft Windows Server

So all we need to do is attach this  task from Windows.Server>library.MP  to  Total CPU Utilization monitor as recovery task .

Here is the XML snippet does the action

Here is the Health Explorer after recovery is attached

Now you can go back and track which processes were  causing high processor utilization on your servers from State Changes in Health Explorer.

Tracking Memory Pressure

Same technique can be applied to track   memory pressure, only problem is we don’t have a built in task to achieve this.  So I have  modified the script which collects CPU cycles so it returns memory consumption for the  processes.

Now  whenever you have  a problem with CPU or Memory utilization you will be able to access the process causing the situation from health explorer.

Attached 3 MPs for Windows 2003 , 2008 & R2 , 2012&R2

DiagnosticTasksMP.zip

You can find more details about Managed availability on following posts;

Lessons from the Datacenter: Managed Availability

Customizing Managed Availability

One of the  problems of this  component/service based approach is now we don’t have the visibility to individual objects like Mailbox Databases or Transport Queues. These are hidden under  specific health sets.

One of my colleagues Russ Slaten  has written an extension management pack to solve the very same issue for Exchange 2010 his posts can be accessed here;

Monitoring Exchange 2010 Database Backup and Mount Status

After using his management pack in many customers I wanted to do the same for Exchange 2013. I have used  his techniques but I had to modify them to work with  my own requirements. I have also added  some extra monitors  rules and added Queue monitoring. So this is what this MP is all about . Lets get into the details ;

Discoveries

This MP will discovery individual Mailbox databases (stand alone  or DAG)  and Transport Queues on Exchange servers.  Discovery is a PowerShell script runs against Exchange server object every 12 hours.

After importing the MP  discovered mailbox databases can be accessed under Mailbox Database Health  view under default Exchange MP folder.

And discovered Queues can be accessed under Transport Queue Health  view .

Monitors

There are 3 monitors targeting to Mailbox database

• Check Database Mount Status – From Russ’s MP sample , checks if DB is mounted or not
• Mailbox Database LastAnyBackup Check : This is  a modified version of 2010 MP. In my script  I check both incremental and full backup and if any backup exist in configured period monitor is healthy.
• Active Preference Check : This monitor will check if database is mounted on Active Preference 1  if database failovers to any other node  monitor will become warning.

For Queues  I have the following monitors configured ;

Let me remind you that all these are also  monitored under Hubtransport Helath Set for the Exchange Server. So based on your requirement you might use this one or the one provided on Exchange MP.

Additional Views

I have also created a  group to include new custom objects as well as the existing one. This group is referenced in  Active Alerts Extended  view. You can use this view to see all alerts generated by Exchange components.

Performance Collection Rules

There are 7 performance collection rules targeting to Mailbox Databases which collects ;

• Database size
• Database Whitespace
• Number of Mailboxes in Database
• Local Mail Flow latency  ( Test-Mailflow )
• Login Latency  (Uses Test-MAPIConnectivity)
• Last full  backup age
• Last incremental backup age

From rules targeting to Mailbox Database   you can disable the ones  not needed in your organization.

There are 10 performance collection rules targeting to Queues .

These rules can be  modified by selecting  Exchange 2013 Transport Queues as target(class).

Cookdown

Cookdown in Operations Manager is the process of utilizing single data source to monitor multiple objects  on server.  On very basics  if you want to monitor 10 individual database on a server   you need to run your script 10 times which is very inefficient and resource consuming. So with cookdown we modify the script in a way that it outputs multiple property bags for each database we want to monitor and  condition detection modules in monitors/rules take care of the rest.

So I tried to utilize this as much as possible. I share the data source  in multiple  rules  so I have cookdown for all instances and for all rules. For database performance collections   all of these   rules actually retrieves data from 2  scripts.  So  if you have 10 mailbox database  in an Exchange server   instead of 10x5 = 50 scripts , I run just one   which then picked up by individual rules and written to database.

The reason I have 2 script is Test mail flow  script has a timeout parameter and takes longer  then the rest of perf collection so this is the only reason I’ve separated it.

To get the benefit from cookdown all these rules are configured run  same time by Synctime setting  in MP they will run every 15 min with Synctime  setting 00:00.

Additional Configuration(Optional)

When I was developing the management pack my customer wanted  the flexibility to  classify health sets. In default configuration we have a two state monitor Healthy and Error . But for most  customers  all these health sets are not equally important.   So My customer wanted to reflect Customer touch point as critical but all others as warning.  We also had a second requirement where we need to use health sets in dashboards  which doesn't work by default. For details please check my other post   How to fix the Health Rollup Problem in Distributed Applications with Exchange 2013 Health Sets.

To overcome all these I’ve duplicated the existing event monitor  and created 2 replicas. One has unhealthy state defined as warning , other as critical.  These 2 monitors are targeted to Availability  for the health sets and disabled by default. There are also 2 groups defined in MP ;

• Critical Health set Objects ,
• Warning Health set Objects

These 2 monitors have overrides defined to enable then using these groups. . All you need to do is disable the existing monitor ,  add the more important health sets to Critical Health sets group and less important ones to warning health sets group .

Update – Script Based Dashboards

I’ve added two script based dashboards under Exchange 2013 Default folder.

Queues

This dashboard will show all transport servers and their queue status

Database Performance Counters

This dashboard will show all mailbox databases and associated performance counters

Update-  16/3/2015 

Mailbox count for the databases was showing max 1000  mailbox because I forget to add -ResultSet Unlimited  switch. I've updated both MP files to show correct mailbox count.

Important !!!

Always test the supplied MP’s in a test environment first , modify them  based on your  needs than move it to production.

Download link

Zip files contains  the base MP Microsoft.Exchange.15.Custom.Extension.xml   and same MP with 2 additional Script based dashboards shown above)  under MPScriptDashboards folder.

Script Dashboard version requires  new visualization MP comes with UR3.

exc2013ExtesionMP.zip

While working with a customer  we tried to create a distributed application for their Exchange 2013 environment. Plan was to create distributed apps for Customer Touch Points and for Key dependencies then configure Service Level Tracking. But no matter what we do distributed application did not rollup the health of the exchange health set.

To demonstrate lets create a basic DA  for Exchange 2013 . I will add some health set instances in two components.

You can get he health set from Logical entity/Application Component/Windows Application Component/Health Set

This is how my distributed app  looks  like in designer…

After creating the distributed app  when you check the diagram   you will see that health will not rollup  from health set to DA components.

If you open the Health Explorer on KeyDependencies  you will see everything is in unmonitored state

And ”Component Group Health Roll-up”  properties….

This is the Dependency Rollup Monitor for the Availability. . Everything looks  normal except that unnamed monitor icon sitting outside the aggregate rollup monitors.

So lets switch to  Authoring pane and open monitors for the Health Set Class

We can see that  event monitor for Exchange 2013  is  targeting the  entity health directly not the Availability rollup selected for dependency monitor. This is why  we are not seeing any health rollup in DA.

To fix it I have copied the  event monitor  from the exchange MP to an unsealed MP  and targeted it to Availability  Rollup.

After importing this MP  I can see new monitor under availability

DA starts to rollup the health properly …..

Now we have duplicate monitors doing the same job. Since its an event monitor  from resource wise it wont have  much impact on agent performance.

From this point on ;

• You can leave both monitors enabled. This will cause duplicate alerts to be generated. You can disable alerting for our monitor  and keep it only for DA health Rollup.
• You can disable the monitor in the original MP and use only the copied one.  This might have undesirable results if Microsoft releases a new version of the MP and changes the configuration for the monitor. 

In my opinion  its better to keep the original one for alerting and copy one for DA rollup only.

If you intend to use only copied one you also need to copy the dependency rollup monitor definitions. These are not included in the attached MP!

P.S. This posts in this blog are provided AS IS  with no warranties….

Log file monitoring  and event collections rules are not  used as much as other features but in some cases  they provide   perfect solution for our monitoring needs.  In Part I we will create the required event collection rule and in Part 2 I’ll guide you on creating a custom report to present the events we collected.

Scenario

Customer is using  a web service to fetch some sensitive data  and wants to track who  is  querying the web service within the organization. Unfortunately this 3rd party web service  does not have any auditing capabilities so we will try to close this gap using OpsMgr.

I will use Orchestrator web service for this demo. same technique can be used to monitor any kind of  website  hosted in IIS.

IIS Logs

Before we dive into details we need to understand  the IIs log  format. IIs logs can be configured from  Internet Information Services Manager.

Here you can select the fields to be included in the log

Sample log file

This is a section from my orchestrator web service log.This is the log entry is created when you  browse a specific  runbook in orchestrator web console.

#Fields: date time s-sitename s-computername s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) sc-status sc-substatus sc-win32-status time-taken

2014-10-12 20:53:19 W3SVC2 AZDEMOSRV1 fe80::b8b0:55b0:4ef:e23a%32 GET /Orchestrator2012/Orchestrator.svc/Jobs() $expand=Runbook,Statistics&$filter=startswith(Runbook%2FPath%2C'%5CSharepoint%5C')%20and%20(RunbookId%20eq%20guid'0b759245-c56e-4f41-83e7-6eded739ed99')&$select=Id,RunbookId……  81 DEMO\volkan fe80::b8b0:55b0:4ef:e23a%32 Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+6.1…….) 200 0 0 9669

OpsMgr Event Collection Rules

Lets  start with creating an event collection rule. Navigate to Authoring pane in OpsMgr console  and create a new Event Collection rule. We will use Generic CV Text Log .

Give it a name  and select target . I’ll use  Windows Operations system as target and  create a disabled rule. We will  create an override to enable it on specific servers.

Specify the directory for the logs, log pattern and separator. In separator area just press space once , we will use space as our separator.

P.S If you have multiple websites in your IIS server make sure you specify the right directory.

Build event Expression Tab lets you to filter the  events using parameters. Before we can do this we need to now which parameter  represents the data we seek.  So lets have a look at he captured event

Event below is generated when we  start a runbook from Orchestrator web console

2014-10-12 19:12:56 W3SVC2 AZDEMOSRV1 fe80::b8b0:55b0:4ef:e23a%32 POST /Orchestrator2012/Orchestrator.svc/Jobs – 81 DEMO\volkan fe80::b8b0:55b0:4ef:e23a%32 Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+6.1…) 201 0 0 1209

When this log is captured by OpsMgr System.ApplicationLog.GenericCSVLog.FilteredEventProvider will place each field under separate parameter as shown below.

<EventData>
  <DataItem type="System.ApplicationLog.GenericLogEntryData" time="2014-10-12T19:13:11.9659052+00:00" sourceHealthServiceId="FA577900-3B84-786B-8E5A-8B235CE22A2B">
    <LogFileDirectory>C:\inetpub\logs\LogFiles\W3SVC2</LogFileDirectory>
    <LogFileType>Generic CSV Log File Format</LogFileType>
    <LogFileName>C:\inetpub\logs\LogFiles\W3SVC2\u_ex141012.log</LogFileName>
    <Params>
      <Param>2014-10-12</Param>
      <Param>19:12:56</Param>
      <Param>W3SVC2</Param>
      <Param>AZDEMOSRV1</Param>
      <Param>fe80::b8b0:55b0:4ef:e23a%32</Param>
      <Param>POST</Param>
      <Param>/Orchestrator2012/Orchestrator.svc/Jobs</Param>
      <Param>-</Param>
      <Param>81</Param>
      <Param>DEMO\volkan</Param>
      <Param>fe80::b8b0:55b0:4ef:e23a%32</Param>
      <Param>Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+6.1…)</Param>
      <Param>201</Param>
      <Param>0</Param>
      <Param>0</Param>
      <Param>1209</Param>
    </Params>
  </DataItem>
</EventData>

Here I’m interested in in

Parameter 6   =   POST   this  filter will only collect  runbook start and stop log entries. When you specify parameters in Event expression you need to refer them as Params/Param[<number>]

After creating the rule I’ve created an override to enable this rule for specific servers

You can also use UNC share for this type rules , for more details check Kevin Holman's blog post

OperationsManagerDW Event Views

I like spending time on SQL management studio so I’ll  review the events from DW. You can use the query below to find the events captured by  Event Collection rules.  All CSV Text Log rules are published  as GenericCSVLog in data warehouse so  we will use this to filter  the captured events.

SELECT        TOP (1000) Event.vEvent.DateTime, vRule.RuleSystemName, Event.vEventDetail.EventData, vEventPublisher.EventPublisherName

FROM            Event.vEvent INNER JOIN
                         Event.vEventDetail ON Event.vEvent.EventOriginId = Event.vEventDetail.EventOriginId INNER JOIN
                         vEventLoggingComputer ON Event.vEvent.LoggingComputerRowId = vEventLoggingComputer.EventLoggingComputerRowId INNER JOIN
                         Event.vEventRule ON Event.vEvent.EventOriginId = Event.vEventRule.EventOriginId INNER JOIN
                         vRule ON Event.vEventRule.RuleRowId = vRule.RuleRowId INNER JOIN
                         vEventPublisher ON Event.vEvent.EventPublisherRowId = vEventPublisher.EventPublisherRowId
WHERE    vEventPublisher.EventPublisherName='GenericCSVLog'
--and    (vRule.RuleSystemName = 'MomUIGeneratedRulea7e8d1fadd4b48afb9bc433dd2a5decf')
ORDER BY Event.vEvent.DateTime DESC

Result;

You can review the events to find the rulesystemname for the rule you created and filter the query further by using the rulesystemname. Since we have created this in OpsMgr console  it has a automatically generated name. Clicking the eventdata will bring you all captured event details.

Now we have to use some XML querying in SQL to get what we need from these events.

SELECT        TOP (100) Event.vEvent.DateTime
,vEventDetail.EventData.value('(/EventData/DataItem/Params/Param)[10]', 'varchar(100)') as Username
,vEventDetail.EventData.value('(/EventData/DataItem/Params/Param)[6]', 'varchar(10)')+' - '+vEventDetail.EventData.value('(/EventData/DataItem/Params/Param)[7]', 'varchar(100)') as Operation
,vEventDetail.EventData.value('(/EventData/DataItem/Params/Param)[4]', 'varchar(100)') as Computername
,vEventDetail.EventData.value('(/EventData/DataItem/Params/Param)[8]', 'varchar(256)') as Details
,SUBSTRING(vEventDetail.EventData.value('(/EventData/DataItem/Params/Param)[8]', 'varchar(256)'),37,36) as JOBID

FROM            Event.vEvent INNER JOIN
                         Event.vEventDetail ON Event.vEvent.EventOriginId = Event.vEventDetail.EventOriginId INNER JOIN
                         vEventLoggingComputer ON Event.vEvent.LoggingComputerRowId = vEventLoggingComputer.EventLoggingComputerRowId INNER JOIN
                         Event.vEventRule ON Event.vEvent.EventOriginId = Event.vEventRule.EventOriginId INNER JOIN
                         vRule ON Event.vEventRule.RuleRowId = vRule.RuleRowId INNER JOIN
                         vEventPublisher ON Event.vEvent.EventPublisherRowId = vEventPublisher.EventPublisherRowId
WHERE    vEventPublisher.EventPublisherName='GenericCSVLog'
and    (vRule.RuleSystemName = 'MomUIGeneratedRulea7e8d1fadd4b48afb9bc433dd2a5decf')
ORDER BY Event.vEvent.DateTime DESC

Result

As you can see I’ve modified my rule to capture special get events as well to show you how to query and filter out XML fields.

SQL Reporting Services Report Access

Same technique can be used to  track  who is accessing the reports in SSRS .  You have to create another rule and use the following expression. first part tracks the report access from Reports   and second part tracks the usage when reports are run from reporting services ( in general SC products use second option to render report within their console)

P.S Check out Enable HTTP logging for SSRS

Eventdata shows the  who has run which report.

In PART 2  we will create  a custom report to display this information in  a user friendly way.