here I go again to help you out . In this post I will walk you through a temporary workaround for the long running Application Performance Monitoring (APM) issue affecting Internet Information Services (IIS) and SharePoint servers. Again, this is a temporary workaround that I am sharing to unblock the upgrade/update/new installation of your System Center Operations Manager (SCOM) environments, where IIS and SharePoint are in use and APM will not be used.

The reason why I call it temporary, is because it does not fix the APM issue. Unfortunately, should you need APM monitoring, this workaround cannot help you out and you must wait for the official solution from the Product Group.

NOTE: I got in touch with the SCOM Product Group and they are already working on a permanent fix which is in line with the below changes. Hence, make sure you install the fix and un-configure them once it is released.

The topic:

As you surely know, there is a long running issue impacting SharePoint servers monitored through SCOM. This issue has been already discussed in several posts like:

• https://blogs.technet.microsoft.com/momteam/2017/03/21/apm-feature-in-scom-2016-agent-may-cause-a-crash-for-the-iis-application-pool-running-under-net-2-0-runtime/
• https://blogs.technet.microsoft.com/momteam/2017/05/31/apm-fix-for-agent-crashing-issue-shipped-in-ur3-is-not-completely-resolved/
• https://blogs.technet.microsoft.com/momteam/2017/06/06/update-on-apm-fix-for-agent-crashing-issue-shipped-in-ur3/
• https://blogs.technet.microsoft.com/kevinholman/2017/08/05/reinstalling-your-scom-agents-with-the-noapm-switch/

Reading through the blog posts listed above, it looks like this issue is only affecting SCOM 2016 agents and newer versions. One of the suggested workaround is to rely on SCOM 2012 R2 agent version. Well, in some cases, this does not work or it's not enough .

In fact, during one of our customer visits, me and my colleague Antonio Canitano ran into a new uncovered scenario causing this known issue. While performing an in-place upgrade of SCOM from version 2012 R2 to version 2016, knowing the existing issue, we left the agents at 2012 R2 version.

Everything seemed to work fine, but after a couple of days the customer started experiencing issues on his SharePoint servers (the same servers which had been working like a charm for 2 years with the same SCOM 2012 R2 agent version) even without having the APM monitoring configured.

The issue:

The problem the customer had, was due to the well-know APM issue discussed several times, but in a different scenario where the failure was supposed to not happen and hence unexpected.

Basically, when an agent is installed with the default settings, the APM binaries are deployed, the APM service is created but left disabled. This means that on the agent machine, the APM stuff is present but not used (dormant state). As you can imagine, in this normal scenario, one never expects to see failures (Application Pool crashes) caused by APM. But you know that theory and reality are 2 different things

Since, as recommended in the official posts above we diligently left the agents at 2012 R2 and hence we did not expect this situation, we had to investigate the real reason why this happened. We only noticed a server restart right after the issue, which made us thinking about some configuration that became effective after the reboot.

We started looking around to see what was changed, but nothing came to mind. Then we started looking at the only thing which was changed: the Management Server and Management Pack versions. In particular we noticed that among the new Management Packs there was the Microsoft.SystemCenter.Apm.Infrastructure that got changed. We compared the 2012 R2 and 2016 versions and among the several differences between there was one in particular that caught our attention.

A new parameter for the rule Apply APM Agent configuration was added: the EnableRTIA Profiler with the default setting of True.

This parameter is responsible for the silent activation of the APM Profiler, which become active after the World Wide Web Publishing Service gets restarted, through the creation of 2 new REG_MULTI_SZ registry keys: one under the W3SVC\Environment path and another one under the WAS\Environment path, both with the same values:

• Reg Key1 == HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W3SVC\Environment
• Reg Key2 == HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\WAS\Environment
• Values ==

    COR_PROFILER={AD5651A8-B5C8-46ca-A11B-E82AEC2B8E78}

    Cor_Enable_Profiling=1

The prove and the repro

It is very easy to repro the issue, if you would like. In a test environment, where SCOM 2016 or 1801 is installed, install the SCOM agent (2012 R2, 2016 or 1801) on a SharePoint server using the default proposed settings. You will see that, as expected, the APM service is set as below.

At this point:

1. Log on to the SharePoint server
2. From an elevated Command Prompt, perform an IISReset
3. Open the SharePoint Central Administration site and see the result:

   

The workaround:

With that amount of information in our hands, we thought about an easy workaround which is broadly applicable. Putting this workaround in place, customers who do not rely on APM could move ahead updating, migrating or simply installing SCOM 2016 or 1801 with no known blockers.

Here are the steps to put the workaround in place:

1. Create an override for the rule Apply APM Agent configuration (yes, the above-mentioned rule) targeted to the .NET Application Monitoring Agent class for the parameter EnableRTIA Profiler that set it to False.

   

2. Wait for the new configuration be delivered and applied. To make sure it is in place, you check for the presence of the following events in the Operations Manager Event Log
   1. Event ID 1201 (containing the name and the version of the management pack you just used to store the override)

      

   2. Event ID 1210

      

3. From an elevated Command Prompt, perform an IISReset

   

4. Check the registry (the 2 keys should have disappeared)

   

5. Open the SharePoint Central Administration site and see the result … BOOOM, back to work !!!:

   

Me and Antonio really hope that with this post will make your update/upgrade or fresh install experience easier relieving you from the need of managing the Agent version. You can safely go ahead with whatever agent version and APM installed.

Today I am going to blog how to resolve event ID 26319 on System Center Operations Manager (SCOM) 1801 related to the p_GetDatawarehouseVersion stored procedure from getting generated.

The topic:

Looking at the Operations Manager event log on Management Server(s) from time to time, as any good SCOM Administrator should do, you will find some instances of the event ID 26319 (yeah, the well-known generic 26319 ).

What will surprise you is that among all these instances, there are some of them which have an interesting description. They report about the lack of permissions to execute a given stored procedure or that a stored procedure is missing. These instances of event 26319 get generated *only* on SCOM 1801 installations, every time the Operations Console is opened.

The issue:

As anticipated above, opening the Operations console will cause the generation of event ID 26319.

This event is generated with 2 different descriptions, depending on what type of installation has been carried out. That means that:

1. On new installations, the event is generated with this string in the description: The EXECUTE permission was denied on the object 'p_GetDatawarehouseVersion', database 'OperationsManagerDW', schema 'sdk'.).

   

2. On upgraded installations (from SCOM 2016), the event is generated with this string in the description: Reason. (Fault Detail is equal to Could not find stored procedure 'sdk.p_GetDatawarehouseVersion'.).

   

The reason behind is that during the installation, the setup procedure for some unknown and strange reasons:

1. Did not correctly set the permission for the sdk database schema on new installations
2. Did not create the stored procedure, and of course did not set the permission, on upgraded installations

The solution:

Luckily, there is a solution for this mess. To resolve the issue and to stop the event ID 26319 from getting generated in both cases, follow the steps below:

1. Logon to the SQL Server host where the SQL Instance hosting the OperationsManagerDW is running
2. Open SQL Server Management Studio and connect to it

   

3. Expand Databases à OperationsManagerDW à Programmability à Stored Procedures

   

4. Scroll down, and search for the p_GetDatawarehouseVersion stored procedure
5. If the stored procedure is already there, meaning that this is a new installation, jump to step 13; otherwise continue to step 6.
6. Download the file attached to this post (taken from a new SCOM 1801 Installation) and save it locally on the SQL Server (i.e. in the C:\Temp folder or on the desktop). The file contains the following code:

   

7. In SQL Server Management Studio, click on File à Open à File

   

8. On the Open File windows, select the file downloaded in step 6 and click Open

   

9. Should you get the following message, click No

   

10. Once the file content is loaded, click Execute

    

11. Make sure the result pane is showing a success message

    

12. After the stored procedure is created, scroll up; right-click on Stored Procedures and click Refresh

    

13. From the stored procedures list, find and select the p_GetDatawarehouseVersion; right-click on it and select Properties.

    

14. In the Properties screen, select the Permission page

    

15. Click on the Search button and in the Select Users or Roles type the OpsMgrReader role name

    

16. Click on Check Names and then click OK
17. In the bottom part of the Permission page, check the box corresponding to Execute in the Grant column and click OK

    

     

The procedure above will fix the issue with the p_GetDatawarehouseVersion stored procedure and will stop the generation of event ID 26319 related to it … and will save you from headache

Thank you all and enjoy problem solving

CREATE_sdk.p_GetDatawarehouseVersion.zip

1. In the Operations Console, create a new scheduled MM entry in SCOM if don't have any.
2. On the primary replica (the active instance), using Microsoft SQL Server management Studio, run the following query against the OperationsManager database to see Scheduled Maintenance mode entries and take note of (at least) one ID from the ScheduleId column
   SELECT * FROM [OperationsManager].[dbo].[MaintenanceModeSchedule]
3. On the primary replica (the active instance), using Microsoft SQL Server management Studio, run the following query against the MSDB and take note of the result:

   SELECT * FROM [msdb].dbo.sysjobs WHERE [name] = '<<Value_of_ScheduledId_field_from_previous_query>>'

4. On one of the secondary replicas (the passive instances), run the same as step 3. You will see no lines returned (as expected).

1. On the primary replica, open Microsoft SQL Server Management Studio and connect to it (you can use the listener name or the instance name as you prefer)
2. Expand SQL Server Agent à Jobs
3. Find the job named as the ScheduledId you got from step 3 in the previous section, right-click on it and select Script Jos as à CREATE To à File

1. Give the file a name and click Save
2. Now, connect to one of the secondary replica
3. Expand Databases à System Databases and select the MSDB database
4. Open the file you saved in step 4 and click Execute (make sure you selected the MSDB database and that its name is appearing in the box on the left, right above Object Explorer)

1. Right-click on Jobs and select Refresh. You will see the newly created job appearing.
2. Repeat the above steps for any affected job on any Secondary replica
3. If you want, you can failover the Availability Group over one of the secondary and run the Scheduled Maintenance Mode.

• Authoring for System Center 2012 - Operations Manager at https://technet.microsoft.com/en-us/library/hh457564(v=sc.12).aspx
• Authoring Tools at https://technet.microsoft.com/en-us/library/hh457569(v=sc.12).aspx
• Authoring Management Packs – the fast and easy way, using Visual Studio??? at https://blogs.technet.microsoft.com/kevinholman/2016/06/04/authoring-management-packs-the-fast-and-easy-way-using-visual-studio/

• The frequency: This needs to be adjusted according to your needs. It can be done through overrides. During my tests, I set it to run once per day (86400 seconds)
  
• The script: I basically used the same sample script that I referenced in my previous post with few small adjustments to add the logging and to use parameters for WorkspaceID and PrimaryKey. Of course, you can replace my sample script with any other script that fits your needs the best; it's just a matter of doing it the right way (parameters order, data retrieving and posting syntax, and so on)
  
• The parameters: They really depends on the script in use; hence parameters can be considered something which are up to your script. In my example, on one of the rules, I did use the following:
  • WorkspaceID
  • PrimaryKey
  • DebugEnabled

  

  And passed them through, this way (you'll need to replace with your WorkspaceID and PrimaryKey):

  

• Take into high consideration the pricing tier you subscribed. Data retention differs according to the pricing tier. For instance, if you are on the free subscription, the data will be kept for only 7; hence configuring the script to run every 2 weeks will put you in a situation in which you have missing data on the week before the collection is configured to run again. More info on pricing and licensing can be found at https://www.microsoft.com/en-us/cloud-platform/operations-management-suite and at https://azure.microsoft.com/en-us/pricing/ .
• Enable the rule only where needed. if you need data from more than one computer, the recommendation is to create a group and to configure the override for that group.
• Disable the debug once you made sure that the rules work as expected.

1. On the top of the script there's an area containing your workspace access information. Here, you have to customize this info before running the script.
   
2. In the center, there's the body or data part, used to retrieve your data. I am going to cover how to change this part in this post.
   
3. In the bottom part, you see the area containing the functions used to sign in and submit the data. I am going to refer to this part as the "fixed part"
   

1. Create the data structure
2. Populate the data structure as needed
3. Convert the data structure to Json

1. Setting the Workspace ID and Primary Key
2. Setting the LogType with my own type (see "My body part" green arrow)
3. Retrieving the data to be sent to OMS (see "My body part" green arrow)
4. Creating and setting the PowerShell object containing the data (see "My body part" green arrow)
5. Converting the PowerShell Object to Json (see "My body part" green arrow)
6. Sending the data (using the Build-Signature and Post-OMSData functions and the Post-OMSData call from the "fixed part")

Running the above script manually, will upload a record with the following format:

This is just a prototype of the script I left to the customer. Hence, it must be considered as an example to explain how to retrieve a registry key to be used as part of a Computer Group Log Search Query like this one:

RegistryKey_CL | where (Value_s == "BR1-Role") | distinct Computer

Lesson learned:

1. LogType: It represents the information type used in the query. Just remind that OMS will add the "_CL" suffix to it.
2. TimeGenerated: This one is not mandatory the default value is the data ingestion timestamp (the timestamp when the data get uploaded).
3. Body part: by appropriately changing the body part you can create a script that uploads different information such as ActiveDirectory attributes (Description, EmployeeID, etc), Active Directory OU location, number of files in a given path and so on.
4. Key fields in the record: Use it if you like to have a more detailed result from your query. Here's why, in this example, I included the computer name.

That's all folks, for now. I will cover how to make the data submission automatic in another post.

Thanks,

Bruno.

HTTPApiCollectorActiveDirectoryOU.zip

HTTPApiCollectorRegKey.zip

I want to share something interesting about the Performance Collection done by the Microsoft Exchange Server 2013 Management Pack.

On one of my customers, we had lots of "Operations Manager failed to start a process" alerts randomly coming from different Exchange Servers. This is a generic alert, indicating that a given script was killed by SCOM itself because it runs over the allowed configured timeout. Even if it is not an issue as per se, a high recurrence of this alert or a high Repeat Count value must not be underestimated since it denotes that the monitoring is not carried on as it should.

As you can see from the screenshot, the failing script was the "NativeCountersCollection.ps1" (which is used by nearly 46 different rules that collects performance data for reporting purpose) that went over the 600 seconds.

For those of you which are not familiar with the Exchange Management Pack, and in particular with this script, it retrieves the configured Exchange counters (for all roles and all instances based on the installed culture) and perform a point-in-time collection using the Get-Counter PowerShell cmdlet. It is normally configured to run every 900 seconds and with a timeout value of 300 seconds.

Since my customer' environment is a medium one with roughly 30 servers and more than 200 mailbox databases, it could have been reasonable to have the script running for more than 300 seconds. As you can imagine there are several instances for each counter, in function of the environment size. So, I started the troubleshooting by increasing the timeout value to 400, then 500 and then 600. At that point, even considering the environment size, I thought that it was not acceptable, in this environment, to have a script that collects point-in-time performance counter values, running for more than 10 minutes.

I went ahead with the troubleshooting by running the script manually (and outside of SCOM) on the impacted servers, taking note of the starting and finishing time. No doubt; the script took around 10 minutes and sometimes even more. Together with the customer, with the assumption that it was not a capacity problem or Exchange related problem since there were neither clear symptoms of CPU, Memory or Disk sufferings nor evident leaks, we decided to investigate the server configuration. We did some basic checks and, surprisingly, we found out that the Power Plan was configured to "Balanced (recommended)"

To prove that this setting could have been the root cause, we changed it to "High Performance" on one server, tested the script once again and . The script execution time went down from the previous 9-10 minutes to roughly 3-4. Yes, you got it right: 60-70% faster. We repeated the test on 3 other servers and same result, so the decision was made: we must change the Power Option setting to use the "High Performance" power plan.

Setting the power plan to High Performance is not negatively impacting the overall system performance, so you can set it on every server. More information about performance issue related to the "Balanced (recommended)" power plan can be found in the article Slow Performance on Windows Server when using the "Balanced" Power Plan at https://support.microsoft.com/en-us/help/2207548/slow-performance-on-windows-server-when-using-the-balanced-power-plan

How can you do it? Well, there are different options:

1. Manually on selected servers only
2. Using a PowerShell script (find a sample for Exchange servers attached below) on selected servers
3. On all server using a Group Policy

I leave the decision on how to change this setting up to you. Nobody knows your infrastructure better than you

Hope that helps.

Thanks!

SettingPowerPlanToHighPerf.ps1.zip

Following the idea of my other post on Active Directory Management Pack – Addendum for Trust Monitoring, I went ahead with the troubleshooting and resolved an issue with the Active Directory Federation Services 2012 R2 that one of my customers is having.

Basically, the issue is that the current MP version, that you can download from HERE, raise an alert saying that the "MEX Endpoint Is Unreachable" even when it is fully OK.

The issue for this false positive seems to be that the script does not exclude from the check, those servers whose role is not "PrimaryComputer". Having that said, I created a modified version of the script (and of whatever is needed to run it) and put it into a simple addendum MP that contains the following:

• A DataSource module which contains the fixes script used to perform the check only on PrimaryServer role
• A UnitMonitorType which parses the output from the DataSource module
• A UnitMonitor which contains the script body and reports on the MEX health by creating an alert in case the status is not good.
• A pre-defined override to disable the original monitor.

This addendum MP, only works with the MP version released by Microsoft for Active Directory Federation Services 2012 R2. To use it, just import it. No need for additional overrides or configuration.

If you want to give it a try, download the Zip file from the link below and import it.

I hope this addendum will relieve your pain, a bit.

Thanks

Microsoft.ActiveDirectoryFederationServices.2012.R2.Addendum.xml

Hi all,

Maybe this has been already blogged, but in the attempt to make IT guys' life easier I want to share a small Power Shell script which I Used to install the necessary Web Console prerequisites for System Center Operations Manager 2016 on Windows Server 2016.

Instead of going back and forth with the System Center Operations Manager prerequisite checker and install the required components one by one, you can use the script in attach which is also screenshotted below:

You can simply launch it from an elevated Windows Power Shell prompt.

The script will ask for an optional parameter, the Source, that will allow the user to specify the path to feature files if the files are not available in the local feature store of the target computer.

This parameter can be left blank or null, hence should you not need to enter any source path, just hit Enter and the script will move forward.

If you want to test it before running in Prod, you can add the -Whatif switch at the end of the Add-WindowsFeature command line and you will get something similar to the screenshot below wit some nice info at the end:

The script will automatically restart the server ONLY if needed (thanks to the restart switch).

My 0.02 :-)

BR1_SCOM2016_IIS_Prerequisites.ps1.zip

• A DataSource module which contains the script used to query and return the status of all existing trusts.
• A UnitMonitorType which parses the output from the DataSource module
• A UnitMonitor which basically reports on the Trust health by creating an alert in case the status is not good.

1. I created a new DataSource that takes another input parameter: the single trust or the comma separated list of trusts
   And which is using a modified version of the script with the exclusion logic
2. Then, because of the new parameter, I had to create a new UnitMonitorType and a new UnitMonitor in order to expose and to pass the new overridable parameter
   
3. Include some pre-defined overrides to disable the original monitor