Event ID 20071 and 21016 are logged on the domain controller, complaining that it cannot communicate to the management server.

Log Name: Operations Manager

Source: OpsMgr Connector

Date: 5/2/2015 2:33:14 PM

Event ID: 20071

Task Category: None

Level: Error

Keywords: Classic

User: N/A

Computer: SG1DC02.contoso.azure

Description:

The OpsMgr Connector connected to OM12R2DEV02.contoso.azure, but the connection was closed immediately without authentication taking place. The most likely cause of this error is a failure to authenticate either this agent or the server . Check the event log on the server and on the agent for events which indicate a failure to authenticate.

Log Name: Operations Manager

Source: OpsMgr Connector

Date: 5/2/2015 2:33:19 PM

Event ID: 21016

Task Category: None

Level: Error

Keywords: Classic

User: N/A

Computer: SG1DC02.contoso.azure

Description:

OpsMgr was unable to set up a communications channel to OM12R2DEV02.contoso.azure and there are no failover hosts. Communication will resume when OM12R2DEV02.contoso.azure is available and communication from this computer is allowed.

Since the management server is actually in the same domain as the domain controller, it doesn't make sense that Kerberos authentication could fail. Checking the Security Event Log on the management server reveals the problem: the domain controller "has not been granted the requested logon type at this machine."

Log Name: Security

Source: Microsoft-Windows-Security-Auditing

Date: 5/2/2015 2:33:19 PM

Event ID: 4625

Task Category: Logon

Level: Information

Keywords: Audit Failure

User: N/A

Computer: OM12R2DEV02.contoso.azure

Description:

An account failed to log on.

Subject:

    Security ID:        NULL SID

    Account Name:        -

    Account Domain:        -

    Logon ID:        0x0

Logon Type:            3

Account For Which Logon Failed:

    Security ID:        NULL SID

    Account Name:        SG1DC02$

    Account Domain:        CONTOSO.AZURE

Failure Information:

    Failure Reason:        The user has not been granted the requested logon type at this machine.

    Status:            0xC000015B

    Sub Status:        0x0

Process Information:

    Caller Process ID:    0x0

    Caller Process Name:    -

Network Information:

    Workstation Name:    -

    Source Network Address:    -

    Source Port:        -

Detailed Authentication Information:

    Logon Process:        Kerberos

    Authentication Package:    Kerberos

    Transited Services:    -

    Package Name (NTLM only):    -

    Key Length:        0

This event is generated when a logon request fails. It is generated on the computer where access was attempted.

The Subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service, or a local process such as Winlogon.exe or Services.exe.

The Logon Type field indicates the kind of logon that was requested. The most common types are 2 (interactive) and 3 (network).

It turns out that the customer has a domain group policy similar as below, which only grants Administrators and Domain Computers group with the user right "Access this computer from the network". Unfortunately, Domain Controllers are not members of Domain Computers group. And it causes the problem you see above.

The solution is simple: Add Domain Controllers group into local administrators group on each Management Servers. The customer will also discuss with his security team if the domain group policy should be modified to include Domain Controllers group in "Access this computer from network".

By the way, here is the default setting of "Access this computer from the network" policy.

Since Microsoft Azure starts supporting SQL AlwaysOn Group Listener with Internal Load Balancing, I can configure a testing environment in Azure to verify the design. The following articles include detailed steps on the SQL AlwaysOn configuration in Azure.

Tutorial: AlwaysOn Availability Groups in Azure (GUI)

Tutorial: Listener Configuration for AlwaysOn Availability Groups

NOTE: The SQL Server virtual machines must be in a separate cloud service from Operations Manager virtual machines. It is because "Azure does not support direct server return with client and server in the same cloud service."

My AlwaysOn Availability Group and Listener are configured successfully, with some steps skipped from the articles, since I already have an Active Directory domain in Azure.

Here are several things I learnt when installing Operations Manager 2012 R2 with SQL AlwayOn.

• Configure RegisterAllProvidersIP and HostRecordTTL setting

  Operations Manager 2012 R2 doesn't support MultiSubnetFailover keyword. So I configured RegisterAllProvidersIP and HostRecordTTL settings using PowerShell script based the article below.

   

  Create or Configure an Availability Group Listener (SQL Server)

  Get-ClusterResource SCAWOCAS | Set-ClusterParameter RegisterAllProvidersIP 0

  Get-ClusterResource SCAWOCAS |Set-ClusterParameter HostRecordTTL 300

  Stop-ClusterResource SCAWOCAS

  Start-ClusterResource SCAWOCAS

   

  NOTE: The SQL Server Availability Group resource must be manually brought online after the above changes.

   

• Configure additional end point ports on Internal Load Balancer (ILB)

  I saw the following error when configuring Operations Manager database.

  

  The solution is to add additional 2 ports (135 and 445) on the ILB. I use the following PowerShell script to add the end point ports.

  # Define variables

  $ServiceName = "sg1omawo" # the name of the cloud service that contains the availability group nodes

  $AGNodes = "sql2012a","sql2012b" # all availability group nodes containing replicas in the same cloud service, separated by commas

  $EndpointName = "omawoendpoint" # name of the endpoint

  $EndpointPorts = "135","445"

   

  $ILBName = "omawoILB" # chosen name for the new ILB

   

  # Configure a load balanced endpoint for each node in $AGNodes using ILB

  ForEach ($node in $AGNodes)

  {

  ForEach ($EndpointPort in $EndpointPorts)

  {

  Get-AzureVM -ServiceName $ServiceName -Name $node | Add-AzureEndpoint -Name "$EndpointName-$EndpointPort" -LBSetName "$EndpointName-$EndpointPort-LB" -Protocol tcp -LocalPort $EndpointPort -PublicPort $EndpointPort -ProbePort 59999 -ProbeProtocol tcp -ProbeIntervalInSeconds 10 -InternalLoadBalancerName $ILBName -DirectServerReturn $true | Update-AzureVM -verbose

  }

  }

  NOTE: The variables $ServiceName, $AGNodes, $EndpointName, and $ILBName should be modified based on settings in your Azure environment.

   

• Configure OperationsManager DB and DW to Availability Group

  After completing the first Management Server installation, the OperationsManager DB and DW needs to be added into Availability Group. The steps are same for both databases:

1. Configure database Recovery model to "Full".
2. Make a full database backup.
3. Add database to Availability Group.

I use "Show Dashboard" from my "OMAWO" Availability Group to verify OperationsManager DB and DW synchronization states.

• Create SQL Login on the Secondary replica SQL Server

  Since Operations Manager 2012 R2 only creates SQL logins on the primary SQL server instance during setup, I have to manually create those SQL Logins on the secondary SQL server instance.

• Configure Transaction Log backup

  I modified the default Availability Group "Backup Preference" setting from "Prefer Secondary" to "Primary", before configuring Transaction Log backup maintenance job.

  

  NOTE: My reason on modifying the setting is to make sure the Transaction Log will always be backed up even the Secondary replica SQL instance is down. You may want to consult your SQL admin on the most suitable "Backup Preference" setting for the Availability Group.

   

   

  I created a maintenance plan to backup transaction log every hour for OperationsManager DB and DW databases. At first I didn't create the maintenance plan. Within one day my Operations Manager stops working. A good lesson learnt.

  

Here is an example how the service is delivered. We simulate several cluster failure scenarios, and use customer’s backup application to recover them from the previous backup. The recover steps are recorded to create a draft Disaster Recovery document. The recovery time is also recorded to calculate RTO (Recovery Time Objective).

The customer had one Windows Server 2008 R2 Failover Cluster incident last year. Someone deleted a computer account from AD. Unfortunately that computer account is a Virtual Computer Object (VCO) used by a SQL Cluster Network Name resource. The SQL Cluster Network Name failed to online during next SQL cluster group failover, which caused the downtime of SQL service. I don’t know what exactly happened at that time, but customer ended up with re-installing the SQL cluster.

It is a bit complex in Windows Server 2008 R2 Failover Cluster to recover a deleted VCO by a Cluster Admin. Basically you need an AD Admin to perform certain steps, if you are not a member of AD Admins. The following two blogs have the detailed explanations and steps.

Recovering a Deleted Cluster Name Object (CNO) in a Windows Server 2008 Failover Cluster

Recovering a Deleted Cluster Name Object (CNO) in a Windows Server 2008 Failover Cluster, Part 2

Start from Windows Server 2012 Failover Cluster, a new feature “Automated repair of cluster virtual computer objects (VCOs) if they are deleted accidentally” is introduced. The network name resource will be brought ONLINE even the associate VCO is deleted from AD. Event ID 1207 is still reported, but now it says:

 “

The computer object associated with the cluster network name resource 'CAP2' could not be updated in domain 'contoso1.local' during the

Resource post online operation.

“

To recover the missing VCO in Windows Server 2012/2012 R2 Failover Cluster, simply offline the network name resource, right click the resource name, then select “More Actions – Repair”. It will create a VCO computer account in AD.

You can use following cmdlet to locate which domain controller is used to creating the VCO.

get-clusterresource <network name resource> | Get-ClusterParameter | where {$_.name -match "CreatingDC"}  

One the domain controller, Event ID 4741 is recorded in the Security Event Log for the VCO creation. 

Since the VCO is a new computer account in AD, any additional SPN (Service Principal Name) registered with the previous VCO will be missing. Here is the SPN list before the VCO deletion. 

Here is the SPN list after repair the VCO. You can see “test/cap2” is missing from the SPN list of the new VCO. 

SPN is required for Kerberos authentication. Although I have never seen one, but in theory, SPN missing will cause Kerberos authentication failure. It may in turn cause client application fails connecting to the cluster resource. As a Cluster Admin, you should notify your AD Admin that you have recovered the VCO, and asked him to re-register any missing SPN. Let the AD Admin to scratch his head to figure out how to find the missing SPN.

Tips:

============

1. For Windows Server 2008 R2 Failover Cluster admins, please make a good friend with your AD admins. You need his/her help to recover the missing VCO so that the corresponding network name resource can be brought online.

    

2. For Windows Server 2012/2012 R2 Failover Cluster admins, please record all VCO’s SPN using “setspn.exe <VCO Computer Name>” command. It will help your AD Admins to quickly re-register all missing SPNs for a completed VCO recovery.

    

Recently I have done a Configuration Manager 2012 R2 engagement to assist customer configuring Software Update feature. The customer wants to use Configuration Manager 2012 R2 to deploy security updates to all production servers. Like most customers originally using WSUS for security updates, this customer wants to have better controls and detailed reports in Configuration Manger 2012 when deploying security updates. One request the customer has is to keep using the existing WSUS server.

The Software Update Point is successfully configured using the existing WSUS server.  But software update scan is failed on all servers. Report "Scan 3 - Clients of a collection reporting a specific state (secondary)" in "Software Updates - D Scan" category shows the reason. The screenshot below is from my testing environment. Hundreds of servers reported the same error “Group policy conflict” in the customer’s environment:

It is a known issue and we need reconfigure the Active Directory Group Policy, "or move client computers to an organizational unit (OU) that does not have this Group Policy setting applied" ("Planning for Software Updates in Configuration Manager - Group Policy Settings for Software Updates"). The customer decides to unlink the WSUS domain Group Policy from the production server OU in Group Policy Management console.

The next day when I arrived customer’s site, the customer told me one critical production server was rebooted at about 3:20am this morning. His boss wanted a full investigation on the cause of the reboot. He asked my assistance to check the production server.  

The investigation is a short one. First we found the server was rebooted due to security updates installation, because System Event Log captured Event ID 22 from “Automatic Updates”. It says: “Restart Required: To complete the installation of the following updates, the computer will be restarted within 15 minutes”   

Then we checked Control Panel\System and Security\Windows Update\Change settings on the production server. The setting is “Install updates automatically (recommended)”, with the installation time set to 3:00 AM.

Other servers are not configured with “Install updates automatically (recommended)” setting. It seems the automatic update setting on this critical production server was configured incorrectly during the Operating System setup.

The content in C:\Windows\WindowsUpdate.log and WSUS Server “Automatic Approvals” options tells the whole story.  

• The production server receives a local group policy to configure the location of WSUS server shortly after Configuration Manager Agent installation. The local group policy only changes the location of the WSUS server using FQDN name.

• The WSUS domain policy is configured to use WSUS server’s NETBIOS name. The automatic updating is configured to “2 – Notify for download and notify for install” in the domain group policy. There will be no automatic update installation with this domain group policy enabled.

• When the WSUS domain policy is unlinked, the production server reverts to its default automatic updates setting, which is “Install updates automatically (recommended)” at 3:00am. The production server will connect to WSUS server using its FQDN name, as configured by local group policy after Configuration Manager Agent installation.

• The Software Update Point configuration DOES NOT change “Automatic Approvals” rule when the same WSUS server is used. The “Default Automatic Approval Rule” is still enabled. It means any production server with “Install updates automatically (recommended)” automatic updates setting, and configured to use this WSUS server, will download and install updates automatically.

Here is the lesson learnt:

===================

The following tasks should be performed before unlinking any WSUS domain group policy:

• Create a new domain group policy to disable local Windows Update setting before unlinking the WSUS domain group policy.

• Create a compliance baseline to verify the new domain group policy has been successfully applied to all servers with Configuration Manager Agent.

From IT Operations perspective, operations such as unlink domain group policy should be done on a test OU with a small number of production servers, before apply the change to all production servers. It will ascertain any unforeseen effects.