UR9 for SCOM 2012 R2 – Step by Step

Article
05/16/2016

This is an updated article replacing the original – to include the deployment of the Linux MP’s which shipped later. Since Microsoft changed blog platforms over to WordPress – it will not allow me to update the previous one.

NOTE: I get this question every time we release an update rollup: ALL SCOM Update Rollups are CUMULATIVE. This means you do not need to apply them in order, you can always just apply the latest update. If you have deployed SCOM 2012R2 and never applied an update rollup – you can go strait to the latest one available. If you applied an older one (such as UR3) you can always go straight to the latest one!

KB Article for OpsMgr: https://support.microsoft.com/en-us/kb/3129774

KB article for Linux updates: https://support.microsoft.com/en-us/kb/3141435

Download catalog site: https://catalog.update.microsoft.com/v7/site/Search.aspx?q=3129774

Key fixes:

SharePoint workflows fail with an access violation under APM
A certain sequence of the events may trigger an access violation in APM code when it tries to read data from the cache during the Application Domain unload. This fix resolves this kind of behavior.
Application Pool worker process crashes under APM with heap corruption
During the Application Domain unload two threads might try to dispose of the same memory block leading to DOUBLE FREE heap corruption. This fix makes sure that memory is disposed of only one time.
Some Application Pool worker processes become unresponsive if many applications are started under APM at the same time
Microsoft Monitoring Agent APM service has a critical section around WMI queries it performs. If a WMI query takes a long time to complete, many worker processes are waiting for the active one to complete the call. Those application pools may become unresponsive, depending on the wait duration. This fix eliminates the need in WMI query and significantly improves the performance of this code path.
MOMAgent cannot validate RunAs Account if only RODC is available
If there's a read-only domain controller (RODC), the MonAgent cannot validate the RunAs account. This fix resolves this issue.
Missing event monitor does not warn within the specified time range in SCOM 2012 R2 the first time after restart
When you create a monitor for a missed event, the first alert takes twice the amount of time specified time in the monitor. This fix resolves the issue, and the alert is generated in the time specified.
SCOM cannot verify the User Account / Password expiration date if it is set by using Password Setting object
Fine grained password policies are stored in a different container from the user object container in Active Directory. This fix resolves the problems in computing resultant set of policy (RSOP) from these containers for a user object.
SLO Detail report displays histogram incorrectly
In some specific scenarios, the representation of the downtime graph is not displayed correctly. This fix resolves this kind of behavior.
APM support for IIS 10 and Windows Server 2016
Support of IIS 10 on Windows Server 2016 is added for the APM feature in System Center 2012 R2 Operations Manager. An additional management pack Microsoft.SystemCenter.Apm.Web.IIS10.mp is required to enable this functionality. This management pack is located in %SystemDrive%\Program Files\System Center 2012 R2\Operations Manager\Server\Management Packs for Update Rollups alongside its dependencies after the installation of Update Rollup 9.
Important Note One dependency is not included in Update Rollup 9 and should be downloaded separately:

Microsoft.Windows.InternetInformationServices.2016.mp
APM Agent Modules workflow fail during workflow shutdown with Null Reference Exception
The Dispose() method of Retry Manager of APM connection workflow is executed two times during the module shutdown. The second try to execute this Dispose() method may cause a Null Reference Exception. This fix makes sure that the Dispose() method can be safely executed one or more times.
AEM Data fills up SCOM Operational database and is never groomed out
If you use SCOM’s Agentless Exception Monitoring to examine application crash data and report on it, the data never grooms out of the SCOM Operational database. The problem with this is that soon the SCOM environment will be overloaded with all the instances and relationships of the applications, error groups, and Windows-based computers, all which are hosted by the management servers. This fix resolves this issue. Additionally, the following management pack’s must be imported in the following order:
- Microsoft.SystemCenter.ClientMonitoring.Library.mp
- Microsoft.SystemCenter.DataWarehouse.Report.Library.mp
- Microsoft.SystemCenter.ClientMonitoring.Views.Internal.mp
- Microsoft.SystemCenter.ClientMonitoring.Internal.mp
The DownTime report from the Availability report does not handle the Business Hours settings
In the downtime report, the downtime table was not considering the business hours. This fix resolves this issue and business hours will be shown based on the specified business hour values.
The updated RDL files are located in the following location:

%SystemDrive%\Program Files\Microsoft System Center 2012 R2\Operations Manager\Server\Reporting

To update the RDL file, follow these steps:
1. Go to https://MachineName/Reports_INSTANCE1/Pages/Folder.aspxMachineName //Reporting Server.
2. On this page, go to the folder to which you want to add the RDL file. In this case, click Microsoft.SystemCenter.DataWarehouse.Report.Library.
3. Upload the new RDL files by clicking the upload button at the top. For more information, see https://msdn.microsoft.com/en-us/library/ms157332.aspx.
Adding a decimal sign in an SLT Collection Rule SLO in the ENU Console on a non-ENU OS does not work
You run the System Center 2012 R2 Operations Manager Console in English on a computer that has the language settings configured to use a non-English (United States) language that uses a comma (,) as the decimal sign instead of a period (.). When you try to create Service Level Tracking, and you want to add a Collection Rule SLO, the value you enter as the threshold cannot be configured by using a decimal sign. This fix resolves the issue.
SCOM Agent issue while logging Operations Management Suite (OMS) communication failure
An issue occurs when OMS communication failures are logged. This fix resolves this issue.

Issues that are fixed in the UNIX and Linux management packs

Discovery of Linux computers may fail for some system locales
Using the Discovery Wizard or Windows PowerShell cmdlets to discover Linux computers may fail during the final Agent Verification step for computers that have some system locales, such as zh_TW.UTF-8. The scxadmin command that is used to restart the agent during the discovery process did not correctly handle Unicode text in the standard out-of-the-service command.
The UNIX/Linux Agent intermittently closes connections during TLS handshaking
Symptoms include the following:
- Failed heartbeats for UNIX or Linux computers, especially when the SSLv3 protocol is disabled on the Management Servers.
- Schannel errors in the System log that contain text that resembles the following:
  
  A fatal error occurred while creating an SSL client credentials. The internal error state is 10013.
- WS-Management errors in the event log that contain text that resembles the following:
  
  WSManFault
  Message = The server certificate on the destination computer (<UNIX/LINUX-COMPUTER-NAME) has the following errors:
  Encountered an internal error in the SSL library.
  Error number: -2147012721 0x80072F8F
  A security error occurred

Lets get started.

From reading the KB article – the order of operations is:

Install the update rollup package on the following server infrastructure:
- Management servers
- Gateway servers
- Web console server role computers
- Operations console role computers
Apply SQL scripts.
Manually import the management packs.
Update Agents

Now, NORMALLY we need to add another step – if we are using Xplat monitoring – need to update the Linux/Unix MP’s and agents. However, in UR8 and UR9 for SCOM 2012 R2, there are no updates for Linux

1. Management Servers

Since there is no RMS anymore, it doesn’t matter which management server I start with. There is no need to begin with whomever holds the RMSe role. I simply make sure I only patch one management server at a time to allow for agent failover without overloading any single management server.

I can apply this update manually via the MSP files, or I can use Windows Update. I have 3 management servers, so I will demonstrate both. I will do the first management server manually. This management server holds 3 roles, and each must be patched: Management Server, Web Console, and Console.

The first thing I do when I download the updates from the catalog, is copy the cab files for my language to a single location:

Then extract the contents:

Once I have the MSP files, I am ready to start applying the update to each server by role.

***Note: You MUST log on to each server role as a Local Administrator, SCOM Admin, AND your account must also have System Administrator (SA) role to the database instances that host your OpsMgr databases.

My first server is a management server, and the web console, and has the OpsMgr console installed, so I copy those update files locally, and execute them per the KB, from an elevated command prompt:

This launches a quick UI which applies the update. It will bounce the SCOM services as well. The update usually does not provide any feedback that it had success or failure.

I got a prompt to restart:

I choose yes and allow the server to restart to complete the update.

You can check the application log for the MsiInstaller events to show completion:

Log Name: Application
Source: MsiInstaller
Date: 1/27/2016 9:37:28 AM
Event ID: 1036
Description:
Windows Installer installed an update. Product Name: System Center Operations Manager 2012 Server. Product Version: 7.1.10226.0. Product Language: 1033. Manufacturer: Microsoft Corporation. Update Name: System Center 2012 R2 Operations Manager UR9 Update Patch. Installation success or error status: 0.

You can also spot check a couple DLL files for the file version attribute.

Next up – run the Web Console update:

This runs much faster. A quick file spot check:

Lastly – install the console update (make sure your console is closed):

A quick file spot check:

Additional Management Servers:

I now move on to my additional management servers, applying the server update, then the console update and web console update where applicable.

On this next management server, I will use the example of Windows Update as opposed to manually installing the MSP files. I check online, and make sure that I have configured Windows Update to give me updates for additional products:

The applicable updates show up under optional – so I tick the boxes and apply these updates.

After a reboot – go back and verify the update was a success by spot checking some file versions like we did above.

Updating Gateways:

I can use Windows Update or manual installation.

The update launches a UI and quickly finishes.

Then I will spot check the DLL’s:

I can also spot-check the \AgentManagement folder, and make sure my agent update files are dropped here correctly:

***NOTE: You can delete any older UR update files from the \AgentManagement directories. The UR’s do not clean these up and they provide no purpose for being present any longer.

2. Apply the SQL Scripts

In the path on your management servers, where you installed/extracted the update, there are two SQL script files:

%SystemDrive%\Program Files\Microsoft System Center 2012 R2\Operations Manager\Server\SQL Script for Update Rollups

(note – your path may vary slightly depending on if you have an upgraded environment of clean install)

First – let’s run the script to update the OperationsManager database. Open a SQL management studio query window, connect it to your Operations Manager database, and then open the script file. Make sure it is pointing to your OperationsManager database, then execute the script.

You should run this script with each UR, even if you ran this on a previous UR. The script body can change so as a best practice always re-run this.

Click the “Execute” button in SQL mgmt. studio. The execution could take a considerable amount of time and you might see a spike in processor utilization on your SQL database server during this operation. I have had customers state this takes from a few minutes to as long as an hour. In MOST cases – you will need to shut down the SDK, Config, and Monitoring Agent (healthservice) on ALL your management servers in order for this to be able to run with success.

You will see the following (or similar) output:

IF YOU GET AN ERROR – STOP! Do not continue. Try re-running the script several times until it completes without errors. In a production environment, you almost certainly have to shut down the services (sdk, config, and healthservice) on your management servers, to break their connection to the databases, to get a successful run.

Technical tidbit: Even if you previously ran this script in UR1, UR2, UR3, UR4, UR5, UR6, UR7, or UR8, you should run this again for UR9, as the script body can change with updated UR’s.

Next, we have a script to run against the warehouse DB. Do not skip this step under any circumstances. From:

%SystemDrive%\Program Files\Microsoft System Center 2012 R2\Operations Manager\Server\SQL Script for Update Rollups

(note – your path may vary slightly depending on if you have an upgraded environment of clean install)

Open a SQL management studio query window, connect it to your OperationsManagerDW database, and then open the script file UR_Datawarehouse.sql. Make sure it is pointing to your OperationsManagerDW database, then execute the script.

If you see a warning about line endings, choose Yes to continue.

You will see the following (or similar) output:

3. Manually import the management packs

There are 55 management packs in this update! Most of these we don’t need – so read carefully.

The path for these is on your management server, after you have installed the “Server” update:

\Program Files\Microsoft System Center 2012 R2\Operations Manager\Server\Management Packs for Update Rollups

However, the majority of them are Advisor/OMS, and language specific. Only import the ones you need, and that are correct for your language. I will remove all the MP’s for other languages (keeping only ENU), and I am left with the following:

What NOT to import:

The Advisor MP’s are only needed if you are using Microsoft Operations Management Suite cloud service, (Previously known as Advisor, and Operation Insights).

The APM MP’s are only needed if you are using the APM feature in SCOM.

Note the APM MP with a red X. This MP requires the IIS MP’s for Windows Server 2016 which are in Technical Preview at the time of this writing. Only import this if you are using APM *and* you need to monitor Windows Server 2016. If so, you will need to download and install the technical preview editions of that MP from https://www.microsoft.com/en-us/download/details.aspx?id=48256

The TFS MP bundle is only used for specific scenarios, such as DevOps scenarios where you have integrated APM with TFS, etc. If you are not currently using these MP’s, there is no need to import or update them. I’d skip this MP import unless you already have these MP’s present in your environment.

However, the Image and Visualization libraries deal with Dashboard updates, and these always need to be updated.

I import all of these shown without issue.

4. Update Agents

Agents should be placed into pending actions by this update for any agent that was not manually installed (remotely manageable = yes):

One the Management servers where I used Windows Update to patch them, their agents did not show up in this list. Only agents where I manually patched their management server showed up in this list. FYI. The experience is NOT the same when using Windows Update vs manual. If yours don’t show up – you can try running the update for that management server again – manually.

If your agents are not placed into pending management – this is generally caused by not running the update from an elevated command prompt, or having manually installed agents which will not be placed into pending.

In this case – my agents that were reporting to a management server that was updated using Windows Update – did NOT place agents into pending. Only the agents reporting to the management server for which I manually executed the patch worked.

I manually re-ran the server MSP file manually on these management servers, from an elevated command prompt, and they all showed up:

You can approve these – which will result in a success message once complete:

Soon you should start to see PatchList getting filled in from the Agents By Version view under Operations Manager monitoring folder in the console:

5. Update Unix/Linux MPs and Agents

The current Linux MP’s can be downloaded from:

https://www.microsoft.com/en-us/download/details.aspx?id=29696

7.5.1050.0 is current at this time for SCOM 2012 R2 and these shipped shortly after UR9.

****Note – take GREAT care when downloading – that you select the correct download for SCOM 2012 R2. You must scroll down in the list and select the MSI for 2012 R2:

Download the MSI and run it. It will extract the MP’s to C:\Program Files (x86)\System Center Management Packs\System Center 2012 R2 Management Packs for Unix and Linux\

Update any MP’s you are already using. These are mine for RHEL, SUSE, and the Universal Linux libraries.

You will likely observe VERY high CPU utilization of your management servers and database server during and immediately following these MP imports. Give it plenty of time to complete the process of the import and MPB deployments.

Next – you need to restart the “Microsoft Monitoring Agent” service on any management servers which manage Linux systems. I don’t know why – but my MP’s never drop/update in the \Program Files\Microsoft System Center 2012 R2\Operations Manager\Server\AgentManagement\UnixAgents\DownloadedKits filder until this servcie is restarted.

Next up – you would upgrade your agents on the Unix/Linux monitored agents. You can now do this straight from the console:

You can input credentials or use existing RunAs accounts if those have enough rights to perform this action.

Finally:

6. Update the remaining deployed consoles

This is an important step. I have consoles deployed around my infrastructure – on my Orchestrator server, SCVMM server, on my personal workstation, on all the other SCOM admins on my team, on a Terminal Server we use as a tools machine, etc. These should all get the matching update version.

Review:

Now at this point, we would check the OpsMgr event logs on our management servers, check for any new or strange alerts coming in, and ensure that there are no issues after the update.

Known issues:

See the existing list of known issues documented in the KB article.

1. Many people are reporting that the SQL script is failing to complete when executed. You should attempt to run this multiple times until it completes without error. You might need to stop the Exchange correlation engine, stop all the SCOM services on the management servers, and/or bounce the SQL server services in order to get a successful completion in a busy management group. The errors reported appear as below:

------------------------------------------------------
(1 row(s) affected)
(1 row(s) affected)
Msg 1205, Level 13, State 56, Line 1
Transaction (Process ID 152) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
Msg 3727, Level 16, State 0, Line 1
Could not drop constraint. See previous errors.
--------------------------------------------------------

Comments

Anonymous
May 17, 2016
Fairly new to SCOM!! Quick question to those looking: Does the account I use to update our SCOM 2012R2 environment have to be in all three groups (Local Admin, SCOM Admin and SA for the databases) to update all the pieces encompassed in UP9? Mine is not a member of the SA role for the databases. I got our DB Team to run the DB scripts to update that portion after I had completed everything up to that. Will everything still update properly using this method? Appreciate any prompt responses to this question!!! Thanks, Tony
- Anonymous
  May 23, 2016
  Our DBAs didn't want to grant me SA rights. In our preprod env they did a SQL trace while I did the upgrade. there was a change that affected the table master so they believe SA rights are needed. I was granted the rights over the weekend to perform the upgrade and then they put my permissions back.
Anonymous
May 17, 2016
@ Tony - I only test it using Local admin, SCOM admin, and SA over DB instance. In the past (SCOM 2007) our updates often required this or they would fail. However, since our updates are now designed to work with Windows Update - which uses local system by default, I don't think we still retain that requirement for SA. The SQL scripts are independent process, so those can be run by your SQL team with their access for sure. The only thing I can think of that might fail - is that we might not place the agents into pending for an agent update, if you don't have SA to SQL, I am not sure of that change is made via SQL or via SDK. I know it fails when using Windows Update, so I suspect it needs access to SQL. Not a big deal either way.
Anonymous
May 25, 2016
Kevin, I would like to know is there any Rollback plan available in case something goes wrong.
- Anonymous
  May 25, 2016
  restore from backup.
  - Anonymous
    May 27, 2016
    Thanks Kevin. Could you please let me know what to backup and how to backup. Also the steps to restore it will be much appreciated.
Anonymous
June 01, 2016
Hi Kevin,can I update of CU5 to CU9 directly without pass by other updates 6,7 and 8 ?Best reagrds
- Anonymous
  June 01, 2016
  If you read the second paragraph of this post:NOTE: I get this question every time we release an update rollup: ALL SCOM Update Rollups are CUMULATIVE. This means you do not need to apply them in order, you can always just apply the latest update. If you have deployed SCOM 2012R2 and never applied an update rollup – you can go strait to the latest one available. If you applied an older one (such as UR3) you can always go straight to the latest one!
  - Anonymous
    June 01, 2016
    Thank's Kevin!
Anonymous
June 28, 2016
Please advise whether it's possible to apply UR9 to an Operations Manager Reporting Server that's installed on it's own server, and so is separate from the Management Server, and is also separate from the SQL back-end.None of the .MSP files included in UR9 appear to be applicable to this server.Microsoft Update also detects no updates required.
- Anonymous
  June 29, 2016
  There are no updates for reporting, that is why the KB article nor my step by step cover this.
Anonymous
July 01, 2016
Brilliant article - my first exposure to SCOM - looking at tidying up and commissioning an incomplete deployment - ran through this exactly as is stated without any issues - granted this was in the test environment with only 2 Man Servers, 1 DB server and 40 agent updates to do but it all counts! Thanks!
Anonymous
July 12, 2016
Hello, KelvinI'm with scom 2012 R2 UR9 and I'm experiencing a problem when I request the report of downtime period, what happens is that the report comes with a table showing the calculated dowtime start and end and other cells do not come with the total period calculated.I wonder if already faced this problem and if there is a solution, detail performed both procedures of the links:http://www.opsconfig.com/troubleshooting-scom-agent-healthy-but-availability-report-for-server-shows-monitoring-unavailable/https://blogs.technet.microsoft.com/kevinholman/2016/05/16/ur9-for-scom-2012-r2-step-by-step-2/I appreciate the helpI posted the case in more detail on TechNet: https://social.technet.microsoft.com/Forums/systemcenter/en-US/95bb4a0d-2290-426d-9ff2-c99a75bf34c6/scom-2012-downtime-availability-report-not-calculate-total-of-downtime-period?forum=operationsmanagergeneral#95bb4a0d-2290-426d-9ff2-c99a75bf34c6Thanks
- Anonymous
  August 22, 2016
  Hello Mr Kevin appreciate the attention the case has already been resolved in TechNet channel, I ask that disconsidere the request.
Anonymous
July 13, 2016
The comment has been removed
Anonymous
August 22, 2016
Hello KevinThanks for the great article.One question that I am trying to sort out is whether or not the agent upgrade will require a reboot for the monitored devices.All of my devices are Windows Computers - servers to be exact (2008 -> 2012 R2).
- Anonymous
  August 22, 2016
  reboot on agents is not required. There are some cases where an agent cannot fully update if a file is locked, so the update of the file is queued until the next reboot. This will be silent to the end user installing the update. So the answer is no - reboot is not required for monitored agents.
Anonymous
August 29, 2016
Seemingly most Images are broken - is it possible to somehow correct that :)
Anonymous
August 31, 2016
Hi Kevin,Love the blog, keep it up! Currently in our production environment we are having issues with AD Event Report generating. Currently, we are trapping numerous events via the SCC Active Directory Audit MP, but once you try to pull that data via the custom Events report, and filter it by Event ID, it comes back with no data, even though we can see its trapping the information in the view, and putting it out as a Warning.We currently have a 2007 and 2012 environment Multi-homed, and the 2007 Custom Event Reports are pulling those events fine. I have double-triple checked everything i could think of. Any ideas what could be causing the Reporting not to pull data?
Anonymous
October 07, 2016
Excellent Article Kevin. We have separate TEST and Production environment at my work place and UR9 upgrade went successful in both the scenarios however one common issue I found is that when running disk capacity view from health explorer for any Windows Server, the console freezes and the only way to regain control is by ending the task in task manager. I was wondering if anyone else has come across this issue and if there is any solution for it. Thanks.
- Anonymous
  October 07, 2016
  I have UR11 here, and cannot repro your issue. Are you sure you patched the console with the UR as well? Does it do this when run from a management server or a client machine running the console?
  - Anonymous
    October 17, 2016
    Thanks Kevin for your reply. Yes Consoles are upgraded to UR9 as well. This is happening on both Mgmt server as well as remote console client machines.
Anonymous
May 16, 2017
The comment has been removed
Anonymous
May 16, 2017
The comment has been removed

Share via

UR9 for SCOM 2012 R2 – Step by Step

Comments

Additional resources