Log Name:      Operations Manager
Source:        Health Service Modules
Date:          8/8/2014 9:19:12 AM
Event ID:      31553
Task Category: Data Warehouse
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      <<Management Server Name>>

Description:
Data was written to the Data Warehouse staging area but processing failed on one of the subsequent operations.

Exception 'SqlException': Sql execution failed. Error 2627, Level 14, State 1, Procedure ManagedEntityChange, Line 368, Message: Violation of UNIQUE KEY constraint 'UN_ManagedEntityProperty_ManagedEntityRowIdFromDAteTime'. Cannot insert duplicate key in object 'dbo.ManagedEntityProperty'. The duplicate key value is (17512, Apr 26 2013  9:50AM).

One or more workflows were affected by this. 
Workflow name: Microsoft.SystemCenter.DataWarehouse.Synchronization.ManagedEntity
Instance name: c12a765d-2be4-404b-93fd-8ebc915d0dcc
Instance ID: {D31B8D2C-1C84-1D9D-D22D-983C985AA572}
Management group: <<MGName>>

The combination of event ID, event source and the specific exception test are important and anything else I talk about only applies to this particular combination.  I say this because event ID 31553 can be raised for numerous exceptions and each one requires different steps.

After reading over this post on SystemCenterCentral.com from Stanislav Zhelyazkov, I was fairly convinced we were up against the same problem and seeing as I didn’t mind losing one or two rows of property changes, I decided to go along with the steps he provided and see if it helped.  Well it did and it didn’t.  It did help, in that we did not see the same exception again, but it didn’t help in that another exception of the same type (same ID, same SQL message, but a different duplicate key value) came up.

In short, we had the problem that Stanislav described, but with an unforeseeable number of rows in the ManagedEntityProptery table.  Ugh!

So what does any of it have to do with getting performance data into the warehouse?  In order to answer that question, I had to do some digging into how the warehouse works and here is what I learned:

• Performance data can’t be inserted into the warehouse for manage entities that don’t exist in the warehouse yet

• Much like the perf or state data sets, the managed entity data is put into a staging table, before it is officially processed into the warehouse ([dbo].[ManagedEntityStage])

• We had a huge backlog of data in the staging table and nothing had been added to the live tables for months

  • SELECT COUNT(*) FROM [dbo].[ManagedEntityStage] WITH(NOLOCK) -- backlog in the stage table

  • SELECT MAX(DWCreatedDateTime) FROM vManagedEntity -- most recent record in the live table

So how does one fix this issue?  Call CSS is the official answer, but you can take Stanislav’s steps and they will work.  The side effect of deleting the row is that the given managed entity row property change will be replaced so if 100% data fidelity is a priority for you, then this is not a process you want to pursue.  If you get into a situation like we did where there are numerous rows that are violating the key constraint though, then you’re signing up for a lot of tedium, so calling CSS is really the best option as they can help and it will get the attention of the System Center Operations Manager product group if this does end up being a fairly common issue that CSS gets called about.

What should we do to avoid this problem in the future?  It’s not clear how to prevent the root problem, but you can look for this problem and react before it gets out of hand.  In short, the process that is responsible for moving the data from the staging to the live table is encountering an unexpected issue that isn’t handled by the stored procedure ([dbo].[ManagedEntityChange]) and as a result the whole process rolls back.  This will happen over and over again and nothing (that I’ve seen at least) outside of manual intervention will fix it.

So…

• Step 1: Set up some monitoring for event ID 31553 on your management servers and look specifically for the “Violation of UNIQUE KEY constraint 'UN_ManagedEntityProperty_ManagedEntityRowIdFromDAteTime'” exception text

• Step 2: If you see this event, check to see how big your ManagedEntityStage backlog is (query above)

• Step 3: If the backlog is small and you have a reliable backup of your warehouse and you can accept the data fidelity concerns this raises, then consider the steps that Stanislav has outlined.  Note: Any manual deletions will work but are entirely not supported so keep that in mind

• Step 4: Call CSS, if budget allows

Cheers,

Cory

Fortunately J.C. Hornbeck has posted a guide on how to get this tracing working even if the tool itself wont work.

Check it out!

https://blogs.technet.com/b/momteam/archive/2013/10/31/how-to-analyze-a-workflow-when-the-opsmgr-workflow-analyzer-doesn-t-work.aspx

Event Data: https://blogs.technet.com/b/drewfs/archive/2013/08/10/event-reporting-with-power-view.aspx

Alert Data: https://blogs.technet.com/b/drewfs/archive/2013/08/19/alert-reporting-with-power-view.aspx

In the previous post I talked about how to get connected to management groups.  In this post I’m going to talk about working with management servers and gateway servers.  I will also introduce you to a script that I wrote, which gives a hierarchical view of state, much like the health explorer in the console.

Following are some useful ways that I’ve found to work with the management and gateway servers via PowerShell:

• Check to see which servers are running which types of roles (e.g. the RMS emulator, the gateway server role, etc.)
• Ensure the base configurations (i.e. action account, version, for your servers are correct
• Check which agents are managed by the server and vice versa
• Make sure all of your servers are healthy and figure what’s wrong with them if they are not

In order to see the cmdlets that you’ll work with in the OperationsManager PowerShell module, for management and gateway servers, run the following command:

get-command -module operationsmanager -Noun *ManagementServer

That will return four cmdlets.  The first two cmdlets (Get-SCOMGatewayManagementServer and Get-SCOMManagementServer) are used to get list of gateway or management servers based.  You can either get the list of all of them for a given MG by providing no parameters or you can use the -ComputerName or –Name parameters to look for servers that have particular names or substrings within their names, respectively.

The first thing to note is that gateway servers are really considered a type of management server in Operations Manager.  As such, if you run the following cmdlet against a management group that has both gateway servers and management servers, you’ll get back a list that contains both types of servers:

Get-SCOMManagementServer

If you pipe the Get-SCOMManagementServer cmdlet into Get-Member you see that is returns objects of type Microsoft.EnterpriseManagement.Administration.ManagementServer.

Get-SCOMManagementServer | get-member

What’s more relevant though is if we look at the members of that class, which includes many properties and a few methods that are relevant to our common needs.  For instance, since the default output of Get-SCOMManagementServer doesn’t distinguish between the two types of servers, we need to take matters into our own hands to figure that out:

Get-SCOMManagementServer | format-table HealthState, Name, Version, ActionAccountIdentity, IsGateway,IsRootManagementServer –AutoSize

And that how I address the first use mentioned above.

The second use is for ensuring the base configurations are set the way we want them to be.  Base configurations, for me, include the following:

• ActionAccountIdentity: If this changes you may not be able to complete agent maintenance in the way you expect.
• AutoApproveManuallyInstallAgents/RejectManuallyInstalledAgents:  I won’t weigh in on which setting is the best, but each team prefers one way or the other so it’s important to confirm it doesn’t change
• MissingHeartbeatThreshold and HeartbeatInterval: The first is the number of heartbeats that can go missing before an agent will be flagged as missing heartbeats in the management group.  The second is the amount of time (seconds) that is allowed between heartbeats.  When a system doesn’t send in a heartbeat for HeartBeatInterval*MissingHeartBeatThreshold seconds the Heartbeat missing monitor gets flipped which can generate alerts.  It also triggers the computer down monitoring.  Set these values too low and you’ll be dealing with a bunch of false positive alerts.  Turn these up too high and you might not know about servers that are offline, soon enough.  Have them set differently on different servers and things can get really confusing.

So, for instance, to detect systems with a bad ActionAccountIdentity, you would do the following:

# Get the action account that should be used
$CorrectRunAs = Get-SCOMRunAsAccount -Name ‘YourAccountNameHere’ | where-object {$_.AccountType -eq 'SCOMActionAccountSecureData'}

# Get the management server(s) that are not using that account
Get-SCOMManagementServer | where-object{$_.ActionAccountIdentity -ne $CorrectRunAs.Name}

Side note:  I tried to programmatically set the default action account (code that doesn’t work below), but apparently the SDK treats the default run as profile as a special case so we can’t do that any other way accept via the OpsMgr console.  I’m sharing this non-working code though in hopes that someone will find it useful.

# This will fail because OpsMgr SDK blocks changes to the default profile
# Wouldn’t it be nice if it worked?  Hopefully this is useful…
foreach ($IncorrectServer in $IncorrectServers) {Set-SCOMRunAsProfile -Instance $IncorrectServer.HostedHealthService -Account $RunAs -Profile $DefaultRunAsProfile}

This brings us to our third use, which is making the connection between agents their management and gateway servers and vice versa.  Here are some commands that I find useful

# Show the list of management servers, their health state and how many agents they manage
get-scommanagementserver | ft -Property Name, HealthState, @{Name='Agent Count'; Expression={$_.GetAgentManagedComputers().Count}} –AutoSize

# Get the list of agents reporting to a server
$Agents = (get-SCOMManagementServer -Name ‘YourServerName’).GetAgentManagedComputers()

# Show just the agents from that server that are not healthy
# Note: Use the HostedHealthService property to look at the real agent state.
$Agents.HostedHealthService | where-object{$_.HealthState -ne 'Success'}

# Given an agent, find the parent management or gateway server(s)
Get-SCOMAgent –Name “YourAgentName” | Get-SCOMParentManagementServer

The last use is making sure all of our servers are healthy and figuring what’s wrong with them if they are not.  The command for getting the unhealthy servers is easy enough:

$UnhealthyServers = get-SCOMManagementServer | where-object{$_.HealthState -ne 'Success'}

The part that’s tricky is getting an understanding of why each server is unhealthy.  This is tricky (or non-obvious at least) for two reasons.  The first is we have to get the hosted health service for each server, which is where the real state of the management server is held.  Doing that is not hard, but it’s also not obvious. 

$UnhealthyServers | foreach-object{$_.HostedHealthService}

The second part is that we need something like the health explorer from the operations console in PowerShell, where we can pass in a health service and get back a hierarchical view of the state of that object.  For that I had to create something. 

The attached PS1 file (Write-SCOMMonitoringStateHierarchy.ps1) contains two functions.  The first function (Write-SCOMMonitoringStateHierarchy) takes in a hierarchy and then uses the second function (Write-SCOMMonitoringState) to write that hierarchy out to the screen.  In order to make the view a bit more useful the functions will display warning and error states as yellow and red respectively.  Also, they do not show monitors that are uninitialized (the empty green circle in the UI).

To use the functions download the PS1 file, save it to some location on your system (“FullPathToFile”) and then do the following:

# load the functions up into your session
. FullPathToFile\Write-SCOMMonitoringStateHierarchy.ps1

# Get the hosted health services, from those get the hierarchies
#  and with those do the output to the screen
$UnhealthyServers | foreach-object{$_.HostedHealthService} | foreach-object{$_.GetMonitoringStateHierarchy()} | Write-SCOMMonitoringStateHierarchy

And that covers the topic of management and gateway servers.  Next time I’ll start working with agents.  The topic of getting agents installed and removed with PowerShell is pretty well covered in other places so we’ll probably breeze over that fast and get into topics around keeping them healthy and well configured.

Cheers until then,
Cory

Write-SCOMMonitoringStateHierarchy.ps1

In the previous post, I introduced the OpsMgr 2012 PowerShell module and discussed a few of the ways that I used to get oriented to it.  Now that we’re oriented, the first thing I want to explore is how connections work.

Both the previous and the current PowerShell implementations for OpsMgr support remotely connecting to one or more management groups.  This is cool for a couple reasons.  The first is I can open up PowerShell on my desktop and run cmdlets without having to TS onto a bunch of servers.  The second is that I can connect to multiple management groups, which comes in use when I want to compare/align two or more environments together or I want to get data from all of my environments at once.

Note: The OpsMgr 2012 module does not support connecting to down-level versions of OpsMgr.

To get familiar with how this works in the OpsMgr 2012 module, run the following command to see the four cmdlets that relate to management group connections.

get-command -module operationsmanager -noun SCOMManagementGroupConnection

Note: You may notice that the SCOMManagementGroupConnection noun (starting with SCOM) is actually an alias for cmdlets that use the SCManagementGroupConnection noun (starting with SC, for system center).  I use the SCOM specific aliases, but either nouns would work the same.

Likewise, there is an about topic that discusses how the connections work.

get-help about_OpsMgr_Connections | more

You could also review the help content provided on each cmdlet to get familiar with them.  With that said, the examples provided only go so deep.

The first step is to create a connection.  As the about topic mentions, there are a couple kinds of connection types to work with; persistent and temporary.  Connections that are created as persistent are held in the state of the PowerShell session, whereas temporary connections are tied to the execution of a specific cmdlet and live only as long as the resulting objects are in state.  The only distinction here is whether you use the New-SCOMManagementGroupConnection to create the connection (persistent) or if you use the -ComputerName (and optionally -Credentials) parameter to get connected (temporary).

We’ll start by walking through persistent connections and then I’ll cover the one use I’m aware of for temporary connections.

Setting up the connection

In order to create a persistent connection, run the New-SCOMManagementGroupConnection cmdlet.

New-SCOMManagementGroupConnection -ComputerName <OpsMgrServerName>

The value of <OpsMgrServerName> can be any OpsMgr 2012 server that is running the OpsMgr SDK service.  Since the SDK service is running on any management server in an OpsMgr 2012 management group, this gives us a lot more flexibility than in 2007 R2, where we could connect to the RMS only.

If the cmdlet produces no output, then you know you did it right.  If you want the cmdlet to return the connection object that it creates, you can include the –Passthru parameter when you run the cmdlet.

If you need to connect to multiple management groups, you keep running the NEW cmdlet for each management group or management server you want to connect to.  This is a big improvement over what you had to do in 2007 R2.

Once your connections are created, you can run the Get-SCOMManagementGroupConnection cmdlet to see the persistent connections that you’ve created.

Get-SCOMManagementGroupConnection

The most important column in the output is the IsActive column.  This bit indicates which connection is the default connection.  The default connection is the connection that will be implicitly used by all cmdlets.  You can run cmdlets against non-active connections, but in those cases you’ll either have to switch which connection is active or pass in the connection details explicitly.  I’ll cover that scenario further down.

Using the connection

In most cases, creating one persistent connection is the first and last thing you’ll do with the connection directly.  I say that because once a persistent connection is established all cmdlets just work, and they run against the connection flagged as “TRUE” for IsActive.

The reason this works is that every cmdlet in the OpsMgr 2012 module includes a parameter named SCSession which takes in an array of type Connection.  If the cmdlet is not explicitly passed a connection object, then it will look in state for a connection with the IsActive bit set to TRUE and it will implicitly work with that.

If you needed to explicitly pass in a connection to a cmdlet you need to first get the connection and then pass it into the cmdlet(s).  For example, you would do the following to get the connection for a specific management group and then use it to get alerts from that management group.

$connection = Get-SCOMManagementGroupConnection –ManagementGroupName “MyGroup”

Get-SCOMAlert –SCSession $connection

An interesting (to me at least) side effect of the way connections work in relation to cmdlets in this module, is that every object that comes out of a cmdlet holds onto its own little connection back to the management group it was pulled from.  You can see this if you run Get-Member, in that there is a ManagementGroup property on almost every object type that can come out of a cmdlet in the OpsMgr module.  That property is of type Microsoft.EnterpriseManagement.ManagementGroup, which is the object type used in the OpsMgr SDK to hold a connection.  You can test this out yourself by doing the following:

## Create the connection
New-SCOMManagementGroupConnection –ComputerName “MyServer”

## Get alerts implicitly using the connection
$alerts = Get-SCOMAlert

## Check out the ManagementGroup property on an alert
$alerts[0] | select-object -Property ManagementGroup | get-member

What this means to you, is that even if the connection that was used in the cmdlet to get the objects gets blown away, the object itself will still be able to interact with the management group, through this mini-connection is carries around.

Working with multiple connections

But what if you have more than one management group you want to work with?  Maybe you’ve got a pre-prod environment that you want to copy settings or management packs from.  Maybe you’ve got scale or security requirements so you needed to deploy more than one MG in order to scale, but you want to see the data from all of those environments in one place.

As I mentioned previously, you create multiple connections by running the New-SCOMManagementGroupConnection cmdlet.  Once you’ve created all of the connections you need you can use the Get-SCOMManagementGroupConnection cmdlet to see them all.

Get-SCOMManagementGroupConnection

The GET cmdlet also has a few parameters for filtering the connections that are returned.

Id:  This is a GUID that uniquely identifies the connection.  The ID column is not shown in the default output so you’ll need to get at it if you need it.  The easiest way to do that is by piping the GET cmdlet to “Select-Object –Property ID”, but any standard PoSH approach will work.  This will come in use when we want to set a different connection as active.

Get-SCOMManagementGroupConnection | select-object -Property ID

Id
--
e67d872b-815f-4d27-a812-a29a9a6fa438
a407c6d6-3feb-452a-aeb0-16d9f50ad44a
6d87b620-447d-40b1-ba68-1745c672676c

ManagementGroupName: Get all connections that exist with the given management group.  This is just a string and it does support typical wildcards.  This comes in use if you have to connect to multiple management groups and you have a predictable MG naming scheme.

Get-SCOMManagementGroupConnection -ManagementGroup "OMCOR2" | format-table ManagementServerName, IsActive –Auto

ManagementServerName IsActive
-------------------- --------
OMServer1            True
OMServer2            False

ComputerName:  Get all connections that exist with the given ComputerName, which would be the name of a management server.

Get-SCOMManagementGroupConnection -ComputerName OMServer1 | format-table ManagementServerName, IsActive –Auto

ManagementServerName IsActive
-------------------- --------
OMServer1            True

If you need to change which connection is active you can get the connection you want and then use the Set-SCOMManagementGroupConnection cmdlet to set it to active.

Get-SCOMManagementGroupConnection –Id e67d872b-815f-4d27-a812-a29a9a6fa438 | Set-SCOMManagementGroupConnection

Note: You can use any parameter you want to on the GET cmdlet, to get a connection, but the ID parameter is the only one that is guaranteed to return one and only one object.

Now that the connections are created you can use them in a couple of ways.

The first way you can use them is to get details from one source management group and then apply those settings to other management groups. 

# Get the source connection
$Source = Get-SCOMManagementGroupConnection -ComputerName “OMServer1”

# Get all other connections
$Target = Get-SCOMManagementGroupConnection | Where-Object{$_.ManagementServerName -ne "OMServer1"}

Now you can do things like the following:

# Sync the Database grooming settings
Get-SCOMDatabaseGroomingSetting -SCSession $Source | Set-SCOMDatabaseGroomingSetting -SCSession $Target

# See which management packs are not the same between MGs
Compare-Object -ReferenceObject (Get-SCOMManagementPack -SCSession $Source) -DifferenceObject (Get-SCOMManagementPack -SCSession $Target)

The second way you can use multiple connections is to get the same set of details from all connections at the same time.

# Get all connections into an object
$Connections = Get-SCOMManagementGroupConnection

# Get alerts from all connections
Get-SCOMAlert -SCSession ($Connections)

# Get all pending management actions and clear them out (Deny)
Get-SCOMPendingManagement -SCSession ($Connections) | Deny-SCOMPendingManagement

Closing connections

If you need to remove a connection you again use the GET cmdlet to get the connection and then you pipe that into the Remove-SCOMManagementGroupConnection cmdlet.

Get-SCOMManagementGroupConnection –Id e67d872b-815f-4d27-a812-a29a9a6fa438 | Remove-SCOMManagementGroupConnection

Note: You can use any parameter you want to on the GET cmdlet, to get a connection, but the ID parameter is the only one that is guaranteed to return one and only one object.  This is especially important in this case since the REMOVE cmdlet will dutifully remove all connections that get passed into it.  You’ve been warned.

Working with temporary connections

Earlier I mentioned there are two types of connections.  The entire post thus far has discussed persistent connections.  The other type of connection is a temporary connection.  Instead of using the NEW cmdlet to create a connection and then implicitly or explicitly using those connections, temporary connections are created when a cmdlet is run with the –ComputerName and –Credential parameters.

I don’t use these at all myself (so I can’t speak from personal experience), but from what I’ve gathered these parameters come in use when you are running a one-liner via a direct call to PowerShell.exe

PowerShell.exe -Command "& {import-module operationsmanager; Get-SCOMAlert -ComputerName OMServer1}"

So that’s what I know about connections.  Next time I get a chance to kick out a blog post I’ll talk about working with management and gateway servers via PowerShell.  I’ll try to dabble into agent topics there too, but I suspect that will merit an entirely separate post.

Cheers until then,
Cory

As some of you may already know the PowerShell support in OpsMgr 2012 changed significantly from OpsMgr 2007 R2. For starters, the cmdlets are no longer an optional part of the UI install.  It used to be you could select whether or not to install the PowerShell interface, when you installed the operations console, but now it installs automatically, every time you install the UI.

Secondly, and more importantly, the implementation is totally different.  This is good in that it was built from scratch for PowerShell 2.0 (and beyond) and it does away with a few of the bad practices that existed in OpsMgr 2007 R2 (i.e. the navigational provider for the instance space).  It is also bad in that much of what we had in the past will not work and in some important ways the behavior has changed for the worse.

In my upcoming posts I will explore the OpsMgr 2012 PowerShell module and share common ways for using the cmdlets.

So let’s get started…

The first step is to import the module.  If you’ve already got PowerShell 3.0 installed then you can skip this step, but let’s cover it for completeness
sake.  Open an instance of PowerShell, and run the following:

import-module operationsmanager

If you get no output and you just see the prompt on the next
line, then everything worked and now the module is loaded.  From there run the following command to see
the “about_” help topics that ship with the module:

get-help OpsMgr

For getting started, I’d recommend reviewing at least the
following:

get-help about_OpsMgr_WhatsNew | more
get-help about_OpsMgr_Cmdlet_Names | more

Note:  As the What’s New topic points out, everything in this module will only work with OpsMgr 2012.  If you want to programmatically work with 2007 R2 then the What’s New topic walks you through how to do that and you’ll need to use your old scripts and one-liners to do it.  All the more reason to get to one version quickly :).

Next, get the full list of cmdlets that are included with the OperationsManager module by running the following:

get-command -module operationsmanager | more

If you find a cmdlet that is of interest to you and you want to learn more about what it does and how to use it, you can go any of the following

get-help <cmdletName> -detailed | more
This will provide the help content for the cmdlet, including the examples.

get-help <cmdletName> -full | more
This will provide the verbose details on the parameters that are available for the cmdlet.

<cmdletName> | Get-Member
This will show you the possible type(s) that a given cmdlet can return and what the methods and properties are on those types.  This is especially useful for figuring out what you can use in pipelining cmdlets together.

In the next post I’ll write about the new way in which the OpsMgr 2012 module handles connections.

Cheers until then,
Cory

I will use this post to summarize all commands, one-liners and short scripts that I identify overtime for PowerShell against OpsMgr 2012

So, in a continuation of the Azure Monitoring stuff I've been working on lately, I see and heard of a lot of places where System Center could be doing a lot of cool stuff to make life better for our application teams that are moving to Azure.  As a result I've submitted a bunch of suggestions to the connect site.  What would be very helpful for me and for the System Center team is if they got more input from all of you in the form of votes on the suggestions that I've submitted.

So, below is the list of the suggestions I've posted specifically around Azure related features Microsoft IT would like to see.  If you look these over and you agree they would be useful for you, please give them an up vote.  If you don't think they're useful, you can down vote too.  If for some reason you can't vote on the connect site, feel free to leave a comment on this post and I can forward that input along.

Thanks in advance!

• Add a way to encrypt storage account connection strings: https://connect.microsoft.com/SC/feedback/details/758265/opsmgr-azure-add-a-supported-way-to-read-an-encrypted-storage-account-connection-string
• Provide monitoring for service bug: https://connect.microsoft.com/SC/feedback/details/758266/opsmgr-azure-provide-monitoring-for-service-bus
• Provide reporting and dashboards for consumption and utilization: https://connect.microsoft.com/SC/feedback/details/758268/opsmgr-azure-provide-reporting-and-dashboards-for-measuring-consumption-and-utilization
• Provide wizards for adding monitoring to azure applications: https://connect.microsoft.com/SC/feedback/details/758269/opsmgr-azure-provide-wizards-for-adding-monitoring-to-azure-applications
• Provide the means to scale role instances up and down based on performance data: https://connect.microsoft.com/SC/feedback/details/758271/opsmgr-azure-provide-the-means-to-detect-the-need-for-scaling-up-or-down-role-instances
• Detect when Azure storage blobs and tables haven't been backed up and give me a simple way to do that backup:  https://connect.microsoft.com/SC/feedback/details/762418/provide-either-a-dpm-or-opsmgr-mp-based-soluiton-to-take-azure-blob-table-backups-snapshots-and-restores
• Detect when my management certs are going to expire and help me easily keep them up to date: https://connect.microsoft.com/SC/feedback/details/762422/provide-a-means-to-detect-when-azure-management-certificates-will-expire-and-a-means-to-refresh-them
• Provide monitoring for shared caching quotas: https://connect.microsoft.com/SC/feedback/details/762424/provide-monitoring-for-shared-caching-quotas
• Provide discovery and monitoring of Azure storage accounts: https://connect.microsoft.com/SC/feedback/details/762426/provide-discovery-and-monitoring-of-windows-azure-storage-accounts
• Provide discovery and monitoring of SQL Azure data syncs: https://connect.microsoft.com/SC/feedback/details/762427/provide-discovery-of-and-montioring-for-sql-azure-data-sync

• Part 1: Enabling instrumentation and monitoring in your Azure app(s)
• Part 2: Setting up the handshake
• Part 3: Discovering your application
• Part 4: When things don’t work

Up until now we’ve worked through getting your Azure application setup for monitoring and we’ve gone through the process of getting it discovered.  But, just like anything else in IT (and life…), something always seems to go wrong the first time.  Also, maybe you’ve setup your Azure application monitoring, but then suddenly, one day you get a call from the team that runs the Azure application and they say, “Hey!  Our Azure app is down and your (insert negative adjective here) monitoring didn’t catch it”.  This post talks about the ways that I’ve seen the Azure MP fail (I’m sure there are some I’ve not hit yet) and talks about how I’ve detected the problem and what I had to do to fix it.

Upfront, I also want to point out that since Azure workflows run on management servers and on watcher nodes the events mentioned in this post manifest in the OperationsManager event log one or both of the following places:

• Any watcher node that is trying to access an Azure application with the certificate
• The root management server in OpsMgr 2007 R2, or one of the management servers in the management server pool in OpsMgr 2012

Your Run As Account(s) are not setup right

In the 2^nd post of this series, we talked about setting up the handshake.  In that we had to setup 2 run as accounts (one to store the certificate and one to store the password used to access the certificate).  I’ve seen this fail in one of two ways.

Way#1: The password used to access the certificate is incorrect:  Unlike windows accounts, where the UI will verify the password you enter against AD, the basic authentication account will gladly take any string you give it.  Its only when OpsMgr tries to use that password on the certificate, that it realizes that something is wrong. 

The way it manifests is in an event in the OperationsManager event log…

Source: Health Service Modules
Event ID: 34000
Description: Failed to load the
certificate for the Windows Azure Service Management API. Please verify that a
valid certificate and password are assigned to the Run As accounts used for
Windows Azure monitoring.
…
Instance name: <InstanceName>
…

And an alert…

Alert Rule: Windows Azure Invalid Certificate
Alert Description: Failed to load
the certificate for the Windows Azure Service Management API. Please verify
that a valid certificate and password are assigned to the Run As accounts used
for Windows Azure monitoring.
…
Instance name: <InstanceName>
…

I included the “Instance name: <InstanceName>” part above, because this is how you figure out which credentials you need to work on.  If you’ve only got one cert or you’ve only got one app, then you don’t have to dig too much, but for those that have multiple accounts to manage, there is some digging you have to do here, and a best practice I suggest:

1. Get the value from <InstanceName>
2. Move to the Monitoring section of the console
3. Navigate to Monitoring -> Windows Azure -> Role Instance State
4. Find the object that has the same name as the <InstanceName> and in the Detail View, note the Full Path Name
5. The format of Full Path Name for an instance is <Hosted Service>\<Deployment>\<Role>\<Instance>, so from this you can take the value from the beginning of the path
6. Now switch to the Authoring section of the console and navigate to Authoring -> Management Pack Templates -> Windows Azure Application
7. Unfortunately, this is where you have to hunt and peck, because the UI does not provide any clear connections between the name of the objects and the name of the Windows Azure Applications. 
   1. Unless, you follow a best practice I suggest.  That best practice is to always put the DNS prefix of the hosted service into the Name of the Windows Azure Application template that you create.  This way, you can use the “Hosted Service” from the full path name above, to quickly make the connection.
   2. Once you find the right Windows Azure Application, open its properties and capture the name of the run as account used under Azure Certificate Password Run As Account
   3. Now switch to the Administration section and navigate to Administration -> Run As Configuration -> Accounts
   4. Locate the run as account that you captured in step 8, above, update its password to the correct value and save the changes.

You know your updates worked by looking in the event log where you first saw the 34000 events.  If you see an event that looks like the following and the value for <HostedServiceName> matches what you saw in step 5 above, then you’re back up and running:

Source:        Health Service Modules
Event ID:      34011
Description: A client for Windows Azure Storage Service has been initialized.

Subscription ID: <SubscriptionIDGuid>
Service name: <HostedServiceName>
…

Way#2: The account(s) are not correctly configured for distribution: In the 2007 R2 timeframe this new feature was added to Run As Accounts for “More Secure” or “Less Secure” distribution.  If you followed the steps in the 2^nd post of this series, then you set things up as “More Secure” and what that means is that you have to explicitly pick which systems the account is authorized to be distributed to.  Unfortunately, if you ever add new management servers or start using a new watcher, then those systems may not be authorized for distribution. This manifests as an event…

Source:        HealthService
Event ID:      1108
Description: An Account specified in the Run As Profile "Microsoft.SystemCenter.Azure.RunAsProfile.Password" cannot be resolved. Specifically, the account is used in the Secure Reference Override   "<SomeLongStringHere>".  This condition may have occurred because the Account is not configured to be distributed to this computer. To resolve this problem, you need to open the Run As Profile specified below, locate the Account entry as specified by its SSID, and either choose to distribute the Account to this computer if appropriate, or change the setting in the Profile so that the target object does not use the specified Account.

Management Group:  <ManagementGroupName>
Run As Profile:  Microsoft.SystemCenter.Azure.RunAsProfile.Password 
…
Object name:  <HostedServiceName>

But no alert!?

So, if you followed the best practice I mentioned above and put the DNS prefix of the hosted service into the Name of the Windows Azure Application template that you created, then you can do the following:

1. Find the right Windows Azure Application, open its properties and capture the names of both run as accounts used (Azure Certificate Password Run As Account and Azure Certificate Run As Account)
2. Now switch to the Administration section and navigate to Administration -> Run As Configuration -> Accounts
3. Locate the run as accounts that you captured in step 1, above.
4. For both accounts do the following:
1. Open the properties of the run as account and switch to the Distribution tab
2. Confirm More secure is selected and click the Add button
3. Add all management servers and all systems that you use as watcher nodes
4. Click OK to apply the changes

Just like the previous case, you know your updates worked by looking in the event log where you first saw the 1108 events.  If you see an event that looks like the following and the value for <HostedServiceName> matches what you saw in the 1108 event, then things are back in order:

Source:        Health Service Modules
Event ID:      34011
Description: A client for Windows Azure Storage Service has been initialized.

Subscription ID: <SubscriptionIDGuid>
Service name: <HostedServiceName>
…

Your watcher node errors out when trying to access the Azure Service Management API

The next group of problems comes when the watcher node (aka proxy agent) tried to reach out and talk to Windows Azure.  For me, I’ve also seen two distinct types of mishaps here.

Way#1: Typos in the Windows Azure Application template: I don’t know about you, but I constantly mistype things.  Half of the reason I take so much time to do blog posts is because I’m fixing typos J.  So, its no surprise that I’ve accidentally mistyped some details into the Windows Azure Application template.  When I do this, it messes up the Azure MP because the Azure MP needs that information to query the Azure Service Management API.

This shows up as an event in the event log…

Source:        Health Service Modules
Event ID:      34004
Description: REST operation against Windows Azure Service has failed because the specified resource in the URI does not exist.

Request method: GET
Request URI: https://management.core.windows.net/<SubscriptionIDGuid you entered>/services/hostedservices/<DNSPrefix you entered>?embed-detail=true
Response status code: NotFound
Response status description: Not Found
Error code: ResourceNotFound
Error message: The hosted service does not exist.
Error XML: <Error xmlns=https://schemas.microsoft.com/windowsazure xmlns:i="https://www.w3.org/2001/XMLSchema-instance"><Code>ResourceNotFound</Code><Message>The hosted service does not exist.</Message></Error>
Exception: The remote server returned an error: (404) Not Found.

But again, no alert!?  In order to track down the typo, you do the following:

1. Capture the <DNSPrefix you entered> from the event
2. Using that string, try full and partial searches of your Windows Azure Application list until you find the right one.
1. Again, this assumes that you’re following the best practice I mentioned above of putting the DNS prefix of the hosted service into the Name of the Windows Azure Application template.
3. Open its properties and reconcile the DNS prefix of the hosted service and the Subscription ID values, with what is in the Azure Portal.
4. Fix any typos and click OK to apply the changes

The same type of 34011 event that is mentioned above, is the clear indicator that you’ve fixed the typo.  Likewise, you can keep refreshing the Hosted Service State view in the monitoring section of the console, until you see the new hosted service show up there.

Way#2: I need to use a proxy server or I typo’d the proxy server: OK, to be honest I’ve not encountered this one, but I know it can happen.  What I’m showing you below is actually from a management group where I just turned off the network connection to simulate a lack of connectivity.  The symptoms should look very similar though.  You would see the following events…

Source:        Health Service Modules
Event ID:      34016
Description:  The remote name could not be resolved: 'management.core.windows.net'. Please check the connection to the remote server or the proxy setting in the workflow configuration.

Subscription ID: <SubscriptionIDGuid>
Service name: <HostedServiceName>
…

or

Source:        Health Service Modules
Event ID:      34016
Description: The remote name could not be resolved: '<DNSPrefix you entered>.table.core.windows.net'. Please check the connection to the remote server or the proxy setting in the workflow configuration.

Subscription ID: <SubscriptionIDGuid>
Service name: <DNSPrefix you entered>
Proxy address: 
Exception: The remote name could not be resolved: '<DNSPrefix you entered>.table.core.windows.net'

And you will see an alert…

Alert Rule: Windows Azure Remote Name Resolution Failure
Alert Description: The remote name could not be resolved: <SomeURL>. Please check the connection to the remote server or the proxy setting in the workflow configuration.

Subscription ID: <SubscriptionIDGuid>
Service name: <DNSPrefix you entered>
Proxy address:
Exception: The remote name could not be resolved: <SomeURL>
…

In order to track down the typo, you do the following:

1. Capture the <DNSPrefix you entered> from the event
2. Using that string, try full and partial searches of your Windows Azure Application list until you find the right one
1. Again, this assumes that you’re following the best practice I mentioned above of putting the DNS prefix of the hosted service into the Name of the Windows Azure Application template
3. Open its properties and reconcile the information entered for the proxy server
4. Fix any typos and click OK to apply the changes

Way#3: Your certificate is not a management certificate on the Azure subscription: Lastly, it is possible that the certificate that is being used in the run as account for Azure monitoring, is not yet added as a management certificate on the hosted service.

If that happens you’ll see the following event…

Source:        Health ServiceModules
Event ID:      34015
Description: Failed to authenticate the request. Please verify that the certificate for the Windows Azure Service Management API is valid and is associated with the subscription.

Subscription ID: <SubscriptionIDGuid>
Service name: <HostedServiceName>

Request method: GET
Request URI: https://management.core.windows.net/<SubscriptionIDGuid>/services/hostedservices/<HostedServiceName>?embed-detail=true
Error code: AuthenticationFailed
Error message: The server failed to authenticate the request. Verify that the certificate is valid and is associated with this subscription.
Error XML: <Error xmlns=https://schemas.microsoft.com/windowsazure xmlns:i="https://www.w3.org/2001/XMLSchema-instance"><Code>AuthenticationFailed</Code><Message>The server failed to authenticate the request. Verify that the certificate is valid and is associated with this subscription.</Message></Error>

Inner exception: The remote server returned an error: (403) Forbidden.
…

And you will see an alert…

Alert Rule: Windows Azure Management Client Authentication Failure
Alert Description: Failed to authenticate the request. Please verify that the certificate for the Windows Azure Service Management API is valid and is associated with the subscription.
Request method: GET
Request URI: https://management.core.windows.net/<SubscriptionIDGuid>/services/hostedservices/<HostedServiceName>?embed-detail=true
Error code: AuthenticationFailed
Error message: The server failed to authenticate the request. Verify that the certificate is valid and is associated with this subscription.
Error XML: <Error xmlns=https://schemas.microsoft.com/windowsazure xmlns:i="https://www.w3.org/2001/XMLSchema-instance"><Code>AuthenticationFailed</Code><Message>The server failed to authenticate the request. Verify that the certificate is valid and is associated with this subscription.</Message></Error>

Inner exception: The remote server returned an error: (403) Forbidden.
…

In order to fix this issue you’ll need to add the certificate as a management certificate, by following the instructions covered in the Monitoring Azure applications – Part 2 post under Add the private key (.CER) to your Azure subscriptions as a “Management Certificate”.

Your watcher node is unable to access the Windows Azure storage account

The last area that I’ve encountered failures is around how the Azure watchers try to access the Windows Azure storage accounts for each host service.  The storage account is where all of the instrumentation is stored for a hosted service, so getting to the account is very important. 

For better or worse, the storage account connection details is not something that you can configure statically within OpsMgr.  Instead the way the Azure MP works is that it uses the service management API to connect to the Hosted Service and then programmatically looks up the connection details to the storage account.  Some of this behavior is also described in the Azure MP guide within the prerequisites section, here.

Following are the two ways in which I’ve seen this part of the process fail:

• Way#1: The name of the setting in the config file is something other than “DiagnosticsConnectionString”
• Way#2: The setting in the config file does not exist or has an invalid value

The resolution steps for both cases is already covered in great detail on Walter Myers’ blog, here.

So that’s it!  Those are all of the ways that I know of that the workflows can fail.  Hopefully this is a useful resource for someone out there on the interwebs.

Cheers!

Check it out here!