The Technical TAM Blog

SAMBA, NTLM SSP, and A Gap in Enterprise Configuration Control

Ted Shelton — Wed, 02 Aug 2017 12:58:23 GMT

Recently I had the opportunity to work with a customer on a Samba upgrade that broke connectivity to a Windows 2008 R2 domain. Background: As reported by the customer, Samba 3.x on their version of Linux was able to connect to any Windows 2008 R2 domain controller, accepting referrals based on IP (\\xxx.yy.zzz) or name (\\). After upgrading to Samba 4.8.4, referrals by name could authenticate to any DC; referrals by IP address would authenticate only to selected domain controllers - they consistently failed against selected DC's. Investigation: Network traces showed that referrals against server names were successfully using Kerberos to authenticate to any domain controller. Referrals by IP address were downgrading to NTLM authentication, and failing against selected DC's. In a failing authentication, the connection would attempt to authenticate with the user's credentials, then with NULL account, then fail with return code Status_Not_Supported (0xc00000bb). Analysis: The results indicated that NTLMv2 authentication could not complete on certain DC's. An evaluation of group policy (gpresult) on each DC showed that the same policies and security settings were being applied to every DC. However, a dump of the registry settings showed differences in configuration between domain controllers. On those DC's where NTLM authentication was failing, the settings for NTLMMinServerSec and NTLMMinClientSec had been set to 0x20080000 (Require 128-bit encryption, Require NTLMv2 session security). On DC's where connections were successful, NTLMMinServerSec and NTLMMinClientSec were set to 0x0 HKEY_LOCAL_MACHINE\System\CurrentControlSet\control\LSA\MSV1_0\NTLMMinServerSec HKEY_LOCAL_MACHINE\System\CurrentControlSet\control\LSA\MSV1_0\NTLMMinClientSec These settings can and should be controlled via group policy, but this was not being done. The associated policies are: Computer Configuration\Windows Settings\Security Settings\Local Policies\Security Options

Network Security: Minimum session security for NTLM SSP based (including secure RPC) servers
Network Security: Minimum session security for NTLM SSP based (including secure RPC) clients

The TechNet article "Changes in NTLM Authentication" (https://technet.microsoft.com/en-us/library/dd566199(WS.10).aspx) describes "two significant changes in the NTLM authentication protocol in Windows 7 and Windows Server 2008 R2". First, NTLM-based minimum session security policy is set to require a minimum of 128-bit encryption for both client computers and servers on new installs (upgrading from a previous OS version retains the setting found during the upgrade). All of the domain controllers were built from scratch, so all of them should have been set to require 128-bit encryption. It is likely that someone in the past had downgraded security on selected DC's for compatibility with some legacy application. Second, Windows Vista and Windows Server 2008 would negotiate NTLM using the computer identity when connecting to earlier versions of Windows. Under Windows 7 and Windows Server 2008 R2, a system service can use either the computer identity or a NULL session. The connections we were seeing from SAMBA would attempt a NULL session and NTLM negotiation would fail. We needed the negotiation to happen using the computer identity, under which both signing and encryption are supported. To configure the servers to successfully negotiate an NTLM connection, we can set the following policy:

Network security: Allow Local System to use computer identity for NTLM

(See https://technet.microsoft.com/en-us/library/jj852275(v=ws.11).aspx for a full discussion of potential settings and implications for this policy). We may have been able to get a successful NTLM connection with the higher security settings by enabling this policy, but the customer opted to eliminate the issue by switching all referrals to use server names. In addition, the investigaton had exposed a gap in enterprise configuration management on selected DC's that were configured to allow NTLM connections with weak security, which was remediated using the appropriate policies.

Service Manager 2016 Console: Error When Launching the Console Using Run As

Ted Shelton — Wed, 02 Aug 2017 12:06:38 GMT

In this customer's situation, running interactively as administrator is prohibited for the usual security reasons. For Service Manager administrative functions, SCSM admins launch the console using Run As and specify administrator credentials. This worked perfectly fine with the SCSM 2012 R2 console, but when they launched the SCSM console using Run As, and then clicked on an incident, they received the results below:

Launch the SCSM 2016 console using Run As and specify an administrator account
Console launches
Select an incident (All Open Incidents)
Open the incident
The incident window opens minimized
Expand the window, which revewals the following error:

Microsoft.EnterpriseManagement.UI.WpfViews.InvalidConfigurationException: Failed to load form based on data in Presentation/Form section of FormView configuration: AssemblyName = Microsoft.EnterpriseManagement.ServiceManager.Applications.ProblemManagement.Forms, TypeName = Microsoft.EnterpriseManagement.ServiceManager.Applications.ProblemManagement.Forms.ProblemForm. at Microsoft.EnterpriseManagement.UI.FormsInfra.FormViewController.CreateFormFromConfiguration (XPathNavigator configNav) at Microsoft.EnterpriseManagement.UI.FormsInfra.FormViewController.ParseConfiguration() at Microsoft.EnterpriseManagement.UI.FormsInfra.FormViewController.EndInit() System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.Windows.Markup.XamlParseException: The invocation of the constructor on type 'Microsoft.EnterpiseManagement.Service Manager.Applications.ProblemManagement.Forms.GeneralTabControl' that matches the specified binding constraints threw an exception. ---> System.UnauthorizedAccessException: Retrieving the COM class factory for component with CLSID {7AB36653-1796-484B-BDFA-E74F1DB7C1DC} failed due to the following error: 80070005 Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED)). at System.Windows.Documents.WinRTSpellerInterop..ctro() at System.Windows.Documents.SpellerInteropBase.CreateInstance() .... As a test, we tried launching the SCSM 2016 console using Run As with a standard user account; we experienced the same error. Note: The issue only occurred when opening incidents, change requests and objects that ship out of the box. The same issue did not occur on custom items that the customer had created (eg, virus items). A registry search for the CLSID mentioned in the error message {7AB36653-1796-484B-BDFA-E74F1DB7C1DC} led to HKLM\SOFTWARE\Classes\CLSID\{7AB36653-1796-484B-BDFA-E74F1DB7C1DC} (Default) REG_SZ Microsoft Spell Checker Factory Class and HKEY_LOCAL_MACHINE\SOFTWARE\Classes\AppID\{7AB36653-1796-484B-BDFA-E74F1DB7C1DC} RunAs REG_SZ Interactive User We also used DCOMCNFG.exe to view the same COM object and confirm the settings. So we could see that the Access Denied is being raised by the system spell checker, which on these machines is configured to run as Interactive User. This is a key finding, since Service Manager 2016 introduces spell check support for editable fields, at least in out of the box work item forms (https://docs.microsoft.com/en-us/system-center/scsm/whats-new#spell-check-in-service-manager-console). Per https://msdn.microsoft.com/en-us/library/windows/desktop/ms680046(v=vs.85).aspx, you can specify the following values for the "RunAs" setting: "Interactive User" "nt authority\localservice" "nt authority\networkservice" "nt authority\system" Note that if you're using DCOMCNFG to manage the COM object, you can change this value to "The launching user", which deletes the RunAs setting in the registry. As a test, we set the value in the registry to "nt authority\localservice". Using this value, we could launch the console using Run As and open incident forms. As a second test, we set the value in DCOMCNFG to "The launching user", verified that the RunAs value disappeared from the registry, and verified that we could open incident forms after launching the console using Run As. Given this information, the customer opted to change the RunAs setting to localservice for SCSM 2016 administrators.

Office 2010 Clients Stop Activating via KMS After Migration to Windows Server 2012 R2

Ted Shelton — Tue, 11 Jul 2017 13:54:26 GMT

Bottom line: Make sure you properly migrate Office KMS activation when you migrate the KMS server. Environment: Windows 7 Enterprise clients with Office Professional Plus 2010 KMS role migrated from Windows 2008 R2 to Windows 2012 R2 over a month before Issue: Customer reported that about 40 days after KMS migration, the installation team found that new Office 2010 clients couldn't activate. On further investigation, we also found that existing Office clients hadn't been able to renew their activation status for the past 40 days. Investigation: - After building the Windows 2012 R2 server and setting up KMS, the customer had installed the Office 2010 Licensing pack but had "experienced errors" - Customer couldn't provide details on what those errors may have been... - We found that the Microsoft Office 2010 KMS Host License Pack version installed on the new Windows 2012 R2 KMS server was a pre-Windows 8/Windows 2012 release that had known issues - We reinstalled using the latest version (14.0.6133.5000, Jan 7 2013) from https://www.microsoft.com/en-us/download/details.aspx?id=25095 - Tried activating new Office 2010 clients and renewing existing clients, but now we were failing with insufficient client count - Noticed in the VAMT tool that multiple clients were registering with the Office 2010 CSVLK instead of GVLK - Checked DNS - found that four workstations had registered themselves as KMS servers - Logged into one of these clients - found that operating system was not activated - Appeared that someone had installed these clients without a license key, then installed Office 2010 and tried to apply the Office 2010 Pro Plus volume license key to these clients in error - This turned out to be the case on all four workstations that had registered themselves as KMS servers, but were in fact not activated to perform KMS function - What appeared to be happening is that clients attempting to activate were locating the workstations registered as KMS servers in DNS, so activation requests were being sent to the wrong host(s) Resolution: - We resolved the matter using the following procedure: On each invalid Windows 7 KMS server, perform the following from an administrative command prompt: 1. Uninstall existing product key: C:\Windows\system32>cscript slmgr.vbs /upk Uninstalled product key successfully. 2. Install the Windows 7 Enterprise GVLK from https://technet.microsoft.com/en-us/library/jj612867(v=ws.11).aspx: C:\Windows\system32>cscript slmgr.vbs /ipk 33PXH-7Y6KF-2VJC9-XBBR8-HVTHH Installed product key 33PXH-7Y6KF-2VJC9-XBBR8-HVTHH successfully. 3. Activate the KMS client using the GVLK: C:\Windows\system32>cscript slmgr.vbs /ato Activating Windows(R) 7, Enterprise edition (aexxxxxx-1bxx-41xx-acxx-6dxxxxxxxxxx) ... Product activated successfully. 4. Change directory to the Office14 folder where ospp.vbs resides: cd c:\program files (x86)\microsoft office\office14 5. Install the Office Professional Plus 2010 GVLK from https://technet.microsoft.com/en-us/library/ee624355(v=office.14).aspx: cscript ospp.vbs /inpkey:VYBBJ-TRJPB-QFQRF-QFT4D-H3GVB ---Processing-------------------------- --------------------------------------- --------------------------------------- ---Exiting----------------------------- 6. Activate Office 2010 using the newly installed GVLK: cscript ospp.vbs /act ---Processing-------------------------- --------------------------------------- Installed product key detected - attempting to activate the following product: SKU ID: 6fxxxxxx-8cxx-41xx-9bxx-83xxxxxxxxxx LICENSE NAME: Office 14, OfficeProPlus-KMS_Client edition LICENSE DESCRIPTION: Office 14, VOLUME_KMSCLIENT channel Last 5 characters of installed product key: H3GVB --------------------------------------- --------------------------------------- ---Exiting----------------------------- 7. Verify that the license activation status now shows that we're activated for the full 180 days: Cscript ospp.vbs /dstatus ---Processing-------------------------- --------------------------------------- SKU ID: 6fxxxxxx-8cxx-41xx-9bxx-83xxxxxxxxxx LICENSE NAME: Office 14, OfficeProPlus-KMS_Client edition LICENSE DESCRIPTION: Office 14, VOLUME_KMSCLIENT channel LICENSE STATUS: ---LICENSED--- Error Code: 0 as licensed Last 5 characters of installed product key: H3GVB REMAINING GRACE: 180 days (259200 minute(s) before expiring) --------------------------------------- --------------------------------------- ---Exiting-----------------------------

Service Manager 2016 Setup Doesn't Detect Data Warehouse Management Server

Ted Shelton — Tue, 09 May 2017 13:28:19 GMT

My client ran into an issue in January running a test upgrade of their lab Service Manager 2012 R2 environment (UR9) to Service Manager 2016. Setup properly detected their management server as a "Service Manager Management Server". Unfortunately, when run on the data warehouse server, it detected that as a "Service Manager Management Server" as well, rather than a "Service Manager Data Warehouse Management Server". They found some other notes about this on the web (see the comments section to Harsh Vardhan Verma's post announcing Service Manager 2016 at https://blogs.technet.microsoft.com/servicemanager/2016/10/25/service-manager-now-available-in-system-center-2016/). After running a number of tests to confirm that the data warehouse server was functioning properly, and trying some thoughts shared by Harsh, the problem persisted. As a test, we ran the setup program against the production data warehouse server - as expected, setup detected that it was running against a "Service Manager Data Warehouse Management Server". Now we had a way to compare good to bad to locate the cause of the issue. Using the SysInternals Process Monitor utility, we collected a log of file and registry accesses by setup.exe on the production data warehouse server. In the SCSMSetupWizard log, we could see that the time logged for the detection of the "Service Manager Data Warehouse Management Server" server. Filtering the ProcMon log on the PID for setup.exe, we zeroed in on the detection time logged in the SCSMSetupWizard log. We could see that setup completed the following steps before recording the Data Warehouse detection to the SCSMSetupWizard log:

It first checked HKCR/Installer/Features/763325B7B0636BE4E94D2D368761DDF0 for the string value SMDW.
It then checked HKLM/Software/Microsoft/Windows/CurrentVersion/Installer/UserData/S-1-5-18/Products/763325B7B0636BE4E94D2D368761DDF0/Features for the string value SMDW.
Finally, it checked HKLM/Software/Microsoft/Windows/CurrentVersion/Installer/UserData/S-1-5-18/Components/ to evaluate a long list of keys.

Running Process Monitor on the lab data warehouse server, and using the time recorded for "Service Manager Management Server" detection in the SCSMSetupWizard log, we found the same series of steps:

It first checked HKCR/Installer/Features/763325B7B0636BE4E94D2D368761DDF0 for the string value SMDW.
It then checked HKLM/Software/Microsoft/Windows/CurrentVersion/Installer/UserData/S-1-5-18/Products/763325B7B0636BE4E94D2D368761DDF0/Features for the string value SMDW.
Finally, it checked HKLM/Software/Microsoft/Windows/CurrentVersion/Installer/UserData/S-1-5-18/Components/ to evaluate a list of keys.

HOWEVER, on the lab data warehouse server improperly detected as a "Service Manager Management Server", the list of components checked was much shorter - maybe half as many component values were checked on the lab server than on the production server. On the lab data warehouse server, in the procmon log we found that component checking stopped after setup looked for the key HKLM/Software/Microsoft/Windows/CurrentVersion/Installer/UserData/S-1-5-18/Components/C136289094C3BF843A93EE1FF3BA42B5 and returned a result "NAME NOT FOUND". At that point setup seems to determine that this must be a "Service Manager Management Server" rather than a "Service Manager Data Warehouse Management Server" server. On the production data warehouse server, setup found this key and continued checking the rest of the components, eventually determining that it was going to upgrade a data warehouse server. On the production server, HKLM/Software/Microsoft/Windows/CurrentVersion/Installer/UserData/S-1-5-18/Components/C136289094C3BF843A93EE1FF3BA42B5 contained the following value: 763325B7B0636BE4E94D2D368761DDF0 REG_SZ 22:\SOFTWARE\Microsoft\System Center\2010\Common\Reporting\Server On both these servers, we checked HKLM\SOFTWARE\Microsoft\System Center\2010\Common\Reporting\Server and verified that each server pointed to the correct reporting server configured for that environment (as expected, since reporting services works properly in each environment). In each case, the reporting server was hosted on a system separate from the Service Manager systems. So the only delta between a data warehouse server properly detected by Service Manager 2016 setup and a data warehouse not detected by setup appeared to be the presence of this key. On the lab data warehouse server, we created the key HKLM/Software/Microsoft/Windows/CurrentVersion/Installer/UserData/S-1-5-18/Components/C136289094C3BF843A93EE1FF3BA42B5. We then created the string value 763325B7B0636BE4E94D2D368761DDF0 under that key, setting it to "22:\SOFTWARE\Microsoft\System Center\2010\Common\Reporting\Server" (without the quotes). We re-ran Service Manager 2016 setup, and now the server was properly detected as the data warehouse server.

The TechTAM Blog on TechNet is Live

Ted Shelton — Sat, 11 Feb 2017 15:52:24 GMT

The Technical TAM blog is now live on Technet. What you can expect: notes from field experience with customers covering a range of topics spanning Service Management, Delivery Management, DevOps, System Center for Infrastructure Management, Private Cloud, and occasional notes on products that might be valuable to a larger audience. Looking forward to sharing and to getting your feedback as well.