The Problem:

Local Publishing of .MSU files by using System Center Updates Publisher (SCUP Version 4.5 or 2011) is not possible. It is not supported by the WSUS product team. Is is not an SCCM issue. The WSUS services is designed to only import and trust .MSU files which are coming from the Microsoft Update Cloud service. And within there are only so-called GDR (General Distribution Release) updates which undergo a more expensive and thorough testing and certification process compared to LDR (Limited Distribution Release) and QFE (Quick Fix Engineering) updates.

However, there are Windows 6.x (Vista, 7, Server 2008, Server 2008 R2) Updates which are shipped in the .MSU format and customers would like to deploy them by using SCCM Software Update Management instead of SCCM Software Distribution.

The Solution:

I managed to setup a functioning workaround as follows:

Step 1: Download the Update you want from Microsoft.com

Step 2: Extract the Updates Contents and locate the .CAB File

To extract the update, use the “expand –F:*” command.

Locate the .cab file of the update:

This cabinet file is actually the update. All other stuff just serves the WUSA.exe and the WSUS detection logic. As we’re rebuilding that stuff in SCUP, we don’t need it anymore. The .CAB file contains the update and is digitally signed – enough for SCUP.

Step 3: Create an .EXE file containing the .CAB file and the DISM installation command:

Yes, we can use the DISM.exe tool to deploy the update by using “online servicing”. The update is in this case not “installed” to the operating system. It is rather “serviced” to the “servicing stack” (CBS, CSI, trusted installer) of the operating system which has been introduced with Windows 6.x. Normally DISM.exe is used to integrate updates into offline .WIM files. But with the /online parameter, it can also be used on a running OS:

dism.exe /online /add-package:<path>Windows6.1-KB917607-x64.cab /quiet /norestart /logpath:c:\917607msu.log /loglevel:2

You might want to test the installation prior to packaging the .EXE

Now bundle both, the .CAB file and the DISM install command into an executable file. Make sure that the file is digitally signed (code signing) and that this signature is trusted (trusted publisher) at the clients:

Step 4: Create a custom update within SCUP

Create a custom update within SCUP and use some simple rules to trigger the update. For Windows 7 update, from my experience, the WIN32_QuickFixEngineering query works fine for all OS updates, even GDR updates:

Step 5: Publish, Synchronize, Deploy, Enjoy

Please Note: Things posted on Blogs are never official Microsoft support statements.

Just a quick note on this case:

If you try to open a SCCM report, you might get an generic HTTP 500 error. Most likely, the problem is tied to the different report limitations in both: SCCM and IIS.

A: The SCCM reporting point has a 10.000 rows limit. This limit can be expanded by changing the registry.You should adjust the registry value on the SMS or SCCM Reporting Point server to allow for returning a larger Row Set.

1. i. You can change the limit by modifying the registry on the reporting point machine.
2. ii. Under the HKEY_LOCAL_MACHINE\Software\Microsoft\SMS\Reporting key, add the DWORD value Rowcount and assign it the value for your desired maximum limit.
3. iii. The maximum row count in decimal is 32767.
4. iv. If you need to return more than 32,767 records, you can set the row count to 0xffffffff hexadecimal, which will return all rows. However, this significantly increases the workload on the SMS site database.

B: The registry change above will allow you to run reports up to 50.000 rows. If you need more, you have to change the ASP buffer size within the IIS server.

1. For IIS6: You need to adjust the AspBufferingLimit setting. The AspBufferingLimit property sets the maximum size of the ASP buffer. The default is approximately 4 MB. To change the default, open %WindowsRoot%\System32\InetSrv\MetaBase.xml and search for “AspBufferingLimit”. Adjust the setting to allow for 1 MB per 1000 records. You can make changes to the MetaBase.xml file while IIS is running only if the edit-while-running feature is enabled. Otherwise, you must stop IIS before editing the MetaBase.xml file.
2. For IIS 7.x: change the setting in the IIS Manager -> ASP -> Limits Properties.

Did you know? You can get more verbose HTTP error messages within your browser is you go to ‘Internet Options’ –> ‘Advanced’ and disable “Show friendly HTTP error messages”. Because “friendly” in most cases means “500”.

Architecture detection using wdsnbp

A SCCM PXE-Handshake with architecture detection uses an additional step. Before sending the final bootstrap file, the SCCM server sends out a bootstrap file which has an included architecture detection. The file is called „wdsnbp.com“. This is necessary, because the architecture information provided by the smbios is not very reliable. The wdsnbp bootstrap file sends back a DHCP request reply which contains an additional DHCP option 250. This high option is normally not used by other tools and contains information concerning the architecture of the client system.

Client pre-boot check and bootfile check

Right after the first contact of the client, die SCCM server checks for the necessary bootfiles. If the bootfiles are not present in the boot-directory (normally Remoteinstall\SMSboot), the server will not send out a DHCP offer. Since the bootfiles are extracted of the WinPE wim files during the startup of the WDS service, you should make sure, that only WinPE WIM files are stored within the SMSPXEIMAGES$ share and not full-os WIM files.

Secondarily the SCCM server checks whether the client exists within the SCCM database. If the client is unknown, the SCCM server will not send out a DHCP offer.

Note, that there has been added an unknown computer support in SCCM 2007 R2:

About Unknown Computer Support for Operating System Deployment

https://technet.microsoft.com/en-us/library/cc431374.aspx

The check of the client can be surveilled within the smspxe.log. Important are the stored procedures „lookupdevice“ and „getbootaction“ The first one checks, whether the client is known while the second one checks, what boot-file should be sent:

· Pxeboot.com (user has to press F12 to proceed – optional Advertisement)
· Pxeboot.n12 (TFTP download starts instantly - mandatory Advertisement)
· Abortpxe.com (stops the PXE Boot)

A SCCM PXE-DHCP handshake looks like that:

DHCP and PXE service run on different machines)

DHCP Discover
· Client broadcast asking for IP address and PXE-capable DHCP service
· DHCP discover package contains option 60
· DHCP discover uses port 67 UDP

SCCM site check
· SCCM server checks, whether client is known (lookupdevice)

DHCP Offer
· Reply of the DHCP server containing IP address offering on port 68 UDP
· Reply of the DHCP or PXE server containing option 60

DHCP Request
· Client broadcast, ready to acknowledge IP address and boot server (port 67 UDP)

DHCP Ack
· Server broadcast, acknowledge clients IP address and lease (port 68 UDP)

DHCP Request
· Client unicast request for options 66 (boot server) and option 67 (boot file) (port 4011 UDP)
· Unicast addresses server which offered option 60

DHCP Ack
· Server ack from PXE server including options 66 and 67 (send WDSNBP)
Start of the bootstrap file “wdsnbp” download using TFTP (UDP 69 and UDP highlevel ports)

DHCP Request
· Client unicast request for options 66 (boot server) and option 67 (boot file) (port 4011 UDP)
· Unicast addresses server which offered option 60
· Contains option 250 with architecture

SCCM site server check
· SCCM server checks for current boot action (getbootaction)

DHCP Ack
· Server ack from PXE server including options 66 and 67 (send pxeboot.n12 for example)

TFTP download of bootfile
TFTP download of WinPE

Screenshot of a sample boot:

A DHCP Handshake runs through the following steps:

DHCP The complete traffic runs on UDP User Datagram Protocol (OSI Layer 4)

Discover
· Client broadcast: asks for IP address
· In the network trace you see a call from 0.0.0.0 to 255.255.255.255
· Runs on port 67 UDP

DHCP Offer
· DHCP server response including IP address for client
· Runs on port 68 UDP

DHCP Request
· Acknowledgement (ack) of the client to the server
· Client requests server ack

DHCP Ack
· Acknowledgement of the IP ownership of the client

Where PXE comes into play:

This DHCP handshake has nothing to do with PXE at this point. PXE stands for pre-boot execution environment and is a rather old technology. It is in-deph defined and documented by Intel:

https://download.intel.com/design/archives/wfm/downloads/pxespec.pdf

PXE does not come with a dedicated boot protocol. It is simply DHCP packets extended with additional DHCP options. It’s formerly known as the bootstrap protocol. If a PXE-enabled network card sends out an DHCP discover package, it will add DHCP option 60, which includes the string “PXEClient:Arch:xxxxx:UNDI:yyyzzz”. Then it waits for DHCP offers.

It will only respond if it gets a DHCP offer including option 60 which means: I am PXE capable and able to send out boot server and boot file information.

The DHCP offer can be splitted into two independent packages, coming from different servers. The DHCP server can send out the DHCP offer containing the clients IP address and the PXE server can send out the DHCP offer containing the option 60.

The PXE client will wait for all needed information up to 60 seconds. DHCP Discover will be retried four times. The four timeouts are 4, 8, 16 and 32 seconds respectively. If the client gets the offer within this timeout, it will proceed with the DHCP request. If not, it will stop with a PXE-MOF error.

A PXE-DHCP Handshake looks like the following:

(DHCP and PXE service run on different machines)

DHCP Discover
· Client broadcast asking for IP address and PXE-capable DHCP service
· DHCP discover package contains option 60
· DHCP discover uses port 67 UDP

DHCP Offer
· Reply of the DHCP server containing IP address offering on port 68 UDP
· Reply of the DHCP or PXE server containing option 60

DHCP Request
· Client broadcast, ready to acknowledge IP address and boot server (port 67 UDP)

DHCP Ack
· Server broadcast, acknowledge clients IP address and lease (port 68 UDP)

DHCP Request
· Client unicast request for options 66 (boot server) and option 67 (bootstrap file) (port 4011 UDP)
· Unicast addresses server which offered option 60

DHCP Ack
· Server ack from PXE server including options 66 and 67

Start of the bootstrap file download using TFTP (UDP 69 and UDP high level ports)

Screenshot: normal PXE boot

You can see in the screenshot that:
· DHCP offers come from two different machines (.2 is the DHCP, .1 is the PXE host)
· The last 2 steps run using unicast

The question is: Is it possible to have an Altiris and an SCCM PXE server within the same network with the target to serve some clients with Altiris and others with SCCM?

The answer is: Yes, but not easy.

The problem is that both systems are sending out custom bootstrap files that have to APIs to hook on. The Altiris server sends out a bootstrap file which ships a gui. The gui then shows a selection menu containing the available bootstrap images. The SCCM server does something quite similar. It sends out a bootstrap file, which detects the target clients architecture (wdsnbp) and then proceeds with an appropriate second bootstrap file.

Thus, what you have to do is to take over control on the FIRST DHCP offer sent to the client.

To accomplish this, you need some sort of decision engine. Technically there are some options:

Option 1: DHCP as decision engine:

You could use DHCP options 66+67 as PXE-referrals if you are able to change them in a dynamic manner based on MAC-addresses. To accomplish this, you would need to import all machines from SCCM and Altiris databases into a new DHCP database. Then, on every DHCP Discover request, you would query the DHCP database for the MAC address and answer the client with the correct DHCP options pointing the client to the appropriate PXE server – either SCCM or Altiris. This contruct requires network programming skills when it comes to use dynamic DHCP options. I’ve implemented it once with a colleague and I can say – it works! It was a custom network daemon attached to a Linux DHCP…

Option 2: Split the clients into dedicated vlans:

Well, this is easy. And it’ll work because DHCP and PXE offer do not need to come from the same machine.

Option 3: Control the PXE server reply behaviour.

If SCCM 2007 does not know a client machine (and does not support unknown computers), it does not send out an DHCP offer. I’m not sure if Altiris supports the same behaviour, but it should. In this scenario, you need to prestage all clients EITHER to Altiris OR to SCCM. Then, on a client boot, the client will query the DHCP and both PXEs and will only get an DHCP offer from the correct PXE server. It is absolutely crucial that you do NOT configure DHCP options in this scenario. If you would, then the DHCP would always answer. I’ve one customer who has implemented this by using two independent SCCM servers.

For RAID Systems:

The alignment topic is only relevant when using stripe-sets of disk for example within a software or hardware raid. What needs to be aligned is the starting location of the partition and the stripe unit boundary in the disk partition that is created on the RAID. If these are not aligned, a volume cluster may be created over a stripe unit boundary instead of next to the stripe unit boundary. This is because Windows uses a factor of 512 bytes to create volume clusters. This behavior causes a misaligned partition. When accessing the volume cluster which is spanning two different stripe units, the IO controller needs to use two I/O – requests instead of one which could lead to a performance impact of 20-40 % on large disk arrays with heavy load (SQL Cluster…)

Not all hardware or software array controllers allow the administrator to choose a stripe unit size; some might provide only a limited set of options. For example, the Windows built-in volume manager for dynamic disks (Dmio.sys) has a fixed stripe unit size of 64 KB. Thus as Windows reserves only the first 63 sectors for the MBR and not the first 64, every 8^th cluster is going the span two stripes: (alignment.jpg)

To work around this issue specify a starting offset of 2,048 sectors (1 megabyte). A starting offset of 2,048 sectors covers most stripe unit size scenarios. This is the default for windows Vista and windows 7 and also adds support for GPT (guid partition tables)

Sources:

You cannot install Windows XP successfully after you use Windows Vista or Windows PE 2.0 to create partitions on a hard disk: https://support.microsoft.com/kb/931760

Disk Subsystem Performance Analysis for Windows: https://www.microsoft.com/whdc/archive/subsys_perf.mspx

Talking of SSDs:

Generally speaking SSDs do have a limited number of possible write cycles. Thus, any application on any operating system may cause issues, if it causes intense write load.  Most of the current crop of SSD’s have decent wear leveling algorithms. As long as you disable the auto (partial) defrag and memory prefetch, then you should be fine for the OS use. 

Windows 7 will add additional benefit to the use of SSDs, as it configures the partition with an disk alignment, that is optimized for best performance. An additionally supports the new “trim” command:

The purpose of Trim Attribute:
File deleting happens in a file system all time, but the information related to the deleted file only kept in OS, not to device. As a result, a device treats both valid data and invalid data(deleted file) in its storage media the same way, all necessary operations keeping data alive are applied. For example, background defect management and error recovery process are used for HDD. Merge, wear leveling and erase are applied to SSD. OS’s process, such as format, Defragger, Recycle bin, temp file deleting and Paging can produce a considerable amount of deleted file. By telling device the deleted file information as invalid data, the device can reduce its internal operation on all invalid data as self-optimization.

Source: https://t13.org/Documents/UploadedDocuments/docs2008/e07154r6-Data_Set_Management_Proposal_for_ATA-ACS2.doc

That means in Windows XP you should

- Disable the defrag

- Disable prefetch https://msdn.microsoft.com/en-us/library/ms940847(WinEmbedded.5).aspx

- Set partition alignment:
NAND flash storage devices are functionally different from traditional hard disk drives in three ways: physical structure, logical structure, and bit erase operation. The logical structure of SSDs differs from tradition hard disk drives because they are built upon pages, which are a given number of bytes (typically 2 KB or 4 KB), and blocks, which are a given number of pages. When the user wants to erase (re-program or write-erase) a given set of bits in any block, the entire block must be erased. This particular anomaly is directly associated with the physics of how NAND works at the die level. If an operating system or data partition is not properly page aligned, there are undesirable consequences. Specifically, if the offset value where the partition begins does not line up with a given page boundary, the SSD flash management system is burdened with extra overhead each time there is a request to perform an erase data operation.

If you wish to use the align parameter in the DiskPart tool, the correct value should be 1 MB for SSDs with page sizes up to 4 KB, and 2 MB for SSDs with a page sizes from 4 KB to 16 KB.

Guidelines for Designing Small Notebook PCs for Windows XP—Version 2.0 - 18

The following is an example of the use of the DiskPart (6.0.6002) tool on a 16-GB SSD with a 4-KB page size, where the align parameter is specified: X:\windows\system32\DiskPart.exe DISKPART> Select Disk 0 DISKPART> Create Partition Primary Align=1024 DISKPART> Exit

2. Install USMT Package onto a client operating system. Please note: The installation on a Server 2008 is not supported and will fail: USMT fails to install on Windows Server https://support.microsoft.com/kb/959066

3. After successful installation, copy the “USMT301” folder within c:\program files to your SCCM Server. In my example it’s volume e:

4. On the USMT package you will find 3 XML files which are used to capture user, system and application settings:

5. These files are used by USMT normally. But as we only need a single folder to capture, we’ll add a additional 4^th file in this case called usmt_temp.xml

6. This XML file will reference only the directory needed on the client:

<?xml version="1.0" encoding="UTF-8"?>

<migration urlid="https://www.microsoft.com/migration/1.0/migxmlext/test">

<component type="Documents" context="System">

<displayName>Component to migrate TK speciffic docs/settings</displayName>

<role role="Data">

<rules>

<include>

<objectSet>

<pattern type="File"> e:\devp\* [*]</pattern>

</objectSet>

</include>

</rules>

</role>

</component>

</migration>

7. Install and Configure a state migration point on the ConfigMgr server if not done already. You need to provide a folder where the clients will temporarily store their captured data:

8. Create a package containing the USMT301 folder. Use defaults for all settings and select “this package contains source files. As source files select the e:\USMT301 folder created previously. Don’t forget to place the package onto the distribution points.

9. Create a custom task sequence containing the steps:

a. Request State Store (select capture state)

b. Capture User State (important! Put in the file-name of the created XML!)

c. Release State Store

10. Place these 3 steps at the very beginning of you task sequence.

11. The files should get captured to your StateMig Point (If you the the error 0x00002024 while requesting the state store, please uninstall update https://support.microsoft.com/kb/974571 and try again. This is a known issue and Microsoft is working on it)

12. To restore the files, add another 3 steps at the end of the task sequence whilst using the “restore” tasks and not the capture tasks.

In order to measure the time needed to install all necessary updates, an administrator needs to know the following information:

1. How many updates will be installed?

2. When have the update installations been started?

3. Is there additional time needed for downloading the update?

4. How long does it take to install specific updates?

5. How can I see that an Update has been installed?

These questions will be discussed in the following chapters. Additionally it will be shown, whether these factors can be measured using a KPI and how.

How many updates will be installed?

The number of updates that are applicable for a Windows XP machine is growing with every month. Each month on patch Tuesday, additional around 3-12 Updates are being released for Windows XP and its components. This means that the number of updates and thus the installation time is increasing with every month. If known updates are being applied to the base image, this number of updates on the other hand is decreasing.

In order to measure the update installation time using a KPI the measuring tool would need to have the information, when the last update of the installation chain has been reached. Also, the reference numbers for measurement have to be adjusted every time an update is added (new update) or removed (update added to base image).

When have the update installations been started?

The start of an update installation process can be triggered by many different tools. If the windows update Agent starts the update installation, a matching start-string is published to the windowsupdate.log. If any other tool starts off the installation, this information may or may not appear in some log file. In any case the beginning of an update installation is not published to the windows event viewer. Only the successful installation is published.

In order to measure the update installation time using a KPI, the start information of the update installation would have to be published by the tool, which triggers the installation. This information is not provided by windows.

Is there additional time needed for downloading the update?

Update, that have to be applied on a system have to be downloaded first normally. If the Windows client is connected to a windows update service, the clients need to download the updates using BITS. If the client is installing the update directly from a share or distribution point (SCCM), the downloads will happen using SMB. The download time of the update is dependent on the following factors:

- Download technology (BITS / SMB)

- Connection Performance (LAN / VPN)

- Update sizes

- Number of updates

- User interference (download disruption, removed cable, hibernating or sleeping clients, unstable wireless connection, unstable UMTS connection etc…)

In order to measure the update installation time using a KPI, the measuring system needs to be aware of these factors that strongly impact the download time.

How long does it take to install specific updates?

Most of the monthly security updates commit a minor change to Windows or the Internet Explorer. Mostly, a dll is getting replaced or some functions are changed or blocked (for example activeX killbits). However, some update commit major changes to the system like installing or changing several updates, fixing .net settings or even updating whole components like .net service pack or a new version of IE. For security reasons, Microsoft will not publish the information on what exactly is changed within an update and what not. It’s fact that every update has a different installation time, because they are committing different changes to the OS or some other component. Additionally, the client’s hardware performance (CPU, Disk), the user behavior (disruption, hibernation, working whilst installing updates,…) and the system configuration (more software installed means longer installation time due to the applicability scanning) impacts the update installation time.

In order to measure the update installation time using a KPI, the measuring system needs to be aware of these factors that strongly impact the installation time.

How can I see that an Update has been installed?

The Bulletin Installation is visible with timestamp within the EventVWR:

If the update has been installed using the windowsupdate agent, the installation succession event is also visible within the event log:

Additionally, on windows XP, a log file with the name of the knowledge base article is created under %windir%:

IF the update was installed using the WindowsUpdate agent, the update will also be shown within the Windows-Update.log:

2010-06-23 01:35:29:373 3280 2a4 Handler :::::::::::::
2010-06-23 01:35:29:373 3280 2a4 Handler :: START :: Handler: Windows Patch Install
2010-06-23 01:35:29:373 3280 2a4 Handler :::::::::
2010-06-23 01:35:29:373 3280 2a4 Handler : Updates to install = 1
2010-06-23 01:35:29:373 3280 2a4 Handler : Installing update {E8714A48-2007-48F5-B7BF-2F33E0C81EDA}.102
2010-06-23 01:35:29:383 3280 e44 Handler Installing with parameters=-q -z -er, sandbox=C:\WINDOWS\SoftwareDistribution\Download\a4c07d9275eb613d842cb1e140d8a426.
2010-06-23 01:35:34:370 896 ba8 Report REPORT EVENT: {A685CFA7-61E7-481D-9734-D54ADF55703D} 2010-06-23 01:35:29:363-0700 1 184 101 {23491AFE-537E-4684-9D10-895516D6FAE7} 104 0 WindowsUpdate Success Content Install Installation successful and restart required for the following update: Security Update for Windows XP (KB980218)
2010-06-23 01:35:36:422 3280 e44 Handler Install completed with 0x0.
2010-06-23 01:35:36:422 3280 2a4 Handler : Install completed: result type = 0x1, installer error = False, error = 0x0, disabled until reboot = No, reboot required = No
2010-06-23 01:35:36:422 3280 2a4 Handler :::::::::
2010-06-23 01:35:36:422 3280 2a4 Handler :: END :: Handler: Windows Patch Install
2010-06-23 01:35:36:422 3280 2a4 Handler :::::::::::::

Opportunities When Measuring The Update Installation Time

Setting Timestamps Using Chained Tasks

In order to measure the update installation time, a chain of tasks would be needed containing a task running right before the update installation and a task running right after the update installation. This chain of tasks would also have to trigger the update installation and collect the installation success messages.

Such a chain of tasks could look like this: (Model 1)

Or like this (Model 2)

Microsoft System Center Configuration Manager supports using such chained tasks by providing the tool “task sequencer”.

Defining a Key Performance Indicator

Summarizing, the update installation time is dependent on the

· number of updates (changes every month)

· installation time of different updates (different on any update)

· download and installation performance

Since it is not possible to exactly predict all of these dependencies, a update installation time key performance indicator would have to be set very broad to allow a lot room for change. Thus, it’s unsure whether a KPI in this case makes sense at all.

Here’s an issue I’ve now seen on two different customers. They’re both running a 3-tier hierarchy with more than 30.000 clients. And both updated from SMS 2003 about half a year ago. The problem they’re experiencing is:

More than 20% of the clients do not report on hardware inventory properly and a lot of DDRs seem to get lost. The Matching errors for the data loader (the hardware inventory processor) and the statsys (the processor of the status messages) on the central site are:

PROBLEM:

sinvproc.log:
The data file "D:\ConfigMgr\inboxes\auth\sinv.box\l5s8o51h.SID" that was submitted by the client whose SMS unique ID is "GUID:34DA8F05-787B-4D3B-8539-AD4322A29E92", was rejected because the file was signed but the authentication key did not match the recorded key for this client.

dataldr.log:
Manager\inboxes\auth\dataldr.box\Process\X3vxse509.MIF" that was submitted by the client whose SMS unique ID is "GUID:1ACBD35E-8426-4CA3-AC84-D1212ACAD3CC", was rejected because the file was signed but the authentication key did not match the recorded key for this client.

It seems like the KeyData, which is stored for every client leads to the issue. The KeyData is the clients authentication key which is used to prove incoming data within inboxes\auth. If the KeyData on the central site somehow gets corrupted, the uploaded data from the clients is being rejected ad you experience backlogs in the folders:

inboxes\auth\dataldr.box\BADMIFS
inboxes\auth\sinv.box\bad
inboxes\auth\statesys.box\corrupt
inboxes\auth\ddm.box\BAD_DDRS

In order to prove the cause, you need to follow these steps:

Step 1: Get the SMS GUID of the suspicious client:

select Name0, smsid0 from system_data where Name0=‘ClientName’

Step 2: Get the KeyData field on both, the central site and the child primary site:

select SMSID, KeyData from ClientKeyData where SMSID=‘GUID:2A3D67C4-0F80-42B9-8659-1F4DE540B91B’

(insert the GUID received on Step 1)

Step 3: Compare the KeyData fields from Step 2:

RESOLUTION:

Currently I’m aware of two workarounds:

1. Delete the entire row of the problem client on the central site server. It will be re-populated within 1-2 minutes.
2. Select the KeyData field from the child primary site server and insert it in the matching field on the central site server. (Be careful, update table commands on the central site server can crash a lot of stuff and are not supported…)

ROOT CAUSE:

Well good question, next question. It looks like it happens when clients are getting re-staged and DDM in Central Site fails to update the table ClientKeyData via
spUpdateClientRegistration.