2781272 - A hotfix is available to add backup support for UEFI-based computers to back up to servers that are running Windows Home Server 2011

After installing this update all was well for a few weeks as far as backing up my new laptop.  At some point though they started to fail and I wasn’t sure why.  It took a while to figure out why and I thought I’d pass along the story in case anyone else runs into a similar issue since my efforts to fix it were in the truest geek fashion – “fun”.  I mean “technically interesting”

A few caveats on what resolved this:

• These steps worked for me and for this specific issue only
• The tools used have the potential to really affect the ability of your computer to work as you want it to (i.e., boot to Windows) so if you’ve never used these tools do your research or ask for help with someone who’s been down this path before
• To be clear; just because I’m blogging this solution doesn’t mean I’m supporting it. Again, these steps worked for me for this issue only. I’m sharing the steps mostly to share the story in case anyone else sees similar behavior with their backup failures since I couldn’t find documentation anywhere else telling me how to fix this

If you’ve used WHS before and are familiar with the Dashboard you’ll know that backup failures are pretty easy to find.  Bringing up the properties of the computer and then seeing why the failure happened may or may not provide useful diagnostic information (at least in my experience with this issue).  I’d already done some testing with the backups and found that failures for me only occurred if I backed-up the C: Drive; other drives I could back up without issue.

With that information in hand I started digging around some more to see if I could find out what was causing this failure.  I logged onto the server itself and in the Event Log found this failure:

Log Name:      WSSG
Source:        Windows Server
Date:          4/27/2013 5:31:06 PM
Event ID:      269
Task Category: Backup
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      WHS
Description:
The Windows Server Client Computer Backup Service received an abort process message from LAPTOP. Reason: 25.

Not very specific is it?

If you dig further and look on the affected client’s Windows Server Event Log you’ll get some additional information that again, isn’t very helpful (IMHO):

Log Name:      WSSG
Source:        Windows Server
Date:          4/27/2013 5:31:10 PM
Event ID:      514
Task Category: Backup
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      laptop
Description:
Backup job XX on WHS did not succeed.  Reason: EspCaptureFailed, System.String[]

I did some digging on the string “Reason: EspCaptureFailed, System.String[]” and only found this one link.  If you look at that link and find anything that means something to you, you’re doing better than me.

At this point I started asking around to see if anyone internally could assist me in figuring out where to go next.  It took some time but eventually I learned that the server backup logs are located in this folder:  %programdata%\microsoft\windows server\logs.  Looking in that folder finds the file called “backup-MMDDYY.log”.  Opening up that file and searching on failure brought me to these lines of interest:

[04/27/2013 17:05:41 15ac] The Windows Server Client Computer Backup Service received an abort process message from laptop.
Reason: 25.
[04/27/2013 17:05:41 15ac] BackupProtocol: Got abort code laptop 1 25
[04/27/2013 17:05:41 15ac] DataFile: current file isD:\ServerFolders\Client Computer Backups\S-1-5-21-863003910-3293894345-3313517167-1041.Machine.configdat and version:: 10
[04/27/2013 17:05:41 15ac] BackupSetOperation:  Backup Failed -  25

Reading through these should get you to the conclusion that there is something on the client causing the backup failure and WHS is off the hook as the suspect.  On the client computer in the same %\ProgramData%\Microsoft\Windows Server\Logs\ path you’ll find the client log (EspCapture.log) with the real issue at hand.  Open up that log file and search on “failed”.  You’ll find that the backup is a robocopy script that logs the parameters used and results.  Scroll down from the timeframe of your failure and you’ll find the real reason on why the backups are failing; here’s a snippet from my log:

     New Dir          3    Z:\EFI\HP\BIOS\New\
        New File             4.0 m    01847.bin
2013/04/27 17:30:47 ERROR 1392 (0x00000570) Copying File Z:\EFI\HP\BIOS\New\01847.bin
The file or directory is corrupted and unreadable.

        New File               256    01847.s12
  0% 
100% 
        New File               256    01847.sig
2013/04/27 17:30:47 ERROR 1392 (0x00000570) Copying File Z:\EFI\HP\BIOS\New\01847.sig
The file or directory is corrupted and unreadable.

As you can see there are corrupt files on the drive causing the backup failure.  Effectively, WHS can’t backup corrupt files resulting in the failure at hand.  So how do I fix it?

The first thing you’re probably thinking is where is the “Z:” drive coming from (well, that’s what I was thinking)?  The short answer is the backup process is a robocopy script that requires a drive letter to work.  The drive letter is assigned on the fly when the backup process starts.  If you look in Windows Explorer or Computer Management you won’t find a “Z:” drive. Another tool called Diskpart though will help us change that so we can fix the corrupted files.  For ease of use I ran all of the following commands from an Administrative Command Prompt (commands I typed are in bold).

C:\Windows\System32>diskpart

Microsoft DiskPart version 6.2.9200

Copyright (C) 1999-2012 Microsoft Corporation.
On computer: LAPTOP

DISKPART> list disk

  Disk ###  Status         Size     Free     Dyn  Gpt
  --------  -------------  -------  -------  ---  ---
  Disk 0    Online          465 GB      0 B        *

DISKPART> list volume

  Volume ###  Ltr  Label        Fs     Type        Size     Status     Info
  ----------  ---  -----------  -----  ----------  -------  ---------  --------
  Volume 0     E                       DVD-ROM         0 B  No Media
  Volume 1     C                NTFS   Partition    440 GB  Healthy    Boot
  Volume 2     D   RECOVERY     NTFS   Partition     24 GB  Healthy
  Volume 3         WINRE        NTFS   Partition    400 MB  Healthy    Hidden
  Volume 4                      FAT32  Partition    260 MB  Healthy    System

Notice that there still is no drive Z: to be found?  Let’s fix that so we can run another utility to fix the corrupt files:

DISKPART> select volume 4

Volume 4 is the selected volume.

DISKPART> assign letter=z

DiskPart successfully assigned the drive letter or mount point.

DISKPART> list volume

  Volume ###  Ltr  Label        Fs     Type        Size     Status     Info
  ----------  ---  -----------  -----  ----------  -------  ---------  --------
  Volume 0     E                       DVD-ROM         0 B  No Media
  Volume 1     C                NTFS   Partition    440 GB  Healthy    Boot
  Volume 2     D   RECOVERY     NTFS   Partition     24 GB  Healthy
  Volume 3         WINRE        NTFS   Partition    400 MB  Healthy    Hidden
* Volume 4     Z                FAT32  Partition    260 MB  Healthy    System

Now that we have a drive letter we can use chkdsk to fix errors on the volume.  First though we need to exit out of diskpart:

DISKPART> exit

Leaving DiskPart...

C:\Windows\System32>z:

FULL DISCLOSURE:  I forgot to take better notes of the actual error being reported by chkdsk /f on my drive so I’m pasting a screen shot instead that was taken by the person who helped me figure this out:

If you go back up on this write-up you’ll find those same file names are the ones that the backup failed because of.  You’ll have to trust me when I say the following step worked for me:

On the Z: drive I ran chkdsk /f which fixed the errors.   Be sure to follow the directions on the screen; failure to do so could render the machine unbootable!  In my case I’d already ran the HP-provided backup utility to reinstall everything if needed.  If you have a similar option I suggest doing that first.

Here’s what my drive looks like running this command today:

Z:\>chkdsk /f
The type of the file system is FAT32.
Cannot lock current drive.

Chkdsk cannot run because the volume is in use by another
process.  Chkdsk may run if this volume is dismounted first.
ALL OPENED HANDLES TO THIS VOLUME WOULD THEN BE INVALID.
Would you like to force a dismount on this volume? (Y/N) y
Volume dismounted.  All opened handles to this volume are now invalid.
Volume Serial Number is 90A0-0AD4
Windows is verifying files and folders...
File and folder verification is complete.

Windows has scanned the file system and found no problems.
No further action is required.

  268,435,456 bytes total disk space.
      110,592 bytes in 5 hidden files.
      630,784 bytes in 154 folders.
  101,720,064 bytes in 413 files.
  165,969,920 bytes available on disk.

        4,096 bytes in each allocation unit.
       65,536 total allocation units on disk.
       40,520 allocation units available on disk.

You’ll want to dismount the Z: drive so below are the steps to do that.  We had to leave diskpart to run the chkdsk /f so going back into diskpart is the first step:

Z:\>c:

C:\Windows\System32>diskpart

Microsoft DiskPart version 6.2.9200

Copyright (C) 1999-2012 Microsoft Corporation.
On computer: LAPTOP

DISKPART> list disk

  Disk ###  Status         Size     Free     Dyn  Gpt
  --------  -------------  -------  -------  ---  ---
  Disk 0    Online          465 GB      0 B        *

DISKPART> list volume

  Volume ###  Ltr  Label        Fs     Type        Size     Status     Info
  ----------  ---  -----------  -----  ----------  -------  ---------  --------
  Volume 0     E                       DVD-ROM         0 B  No Media
  Volume 1     C                NTFS   Partition    440 GB  Healthy    Boot
  Volume 2     D   RECOVERY     NTFS   Partition     24 GB  Healthy
  Volume 3         WINRE        NTFS   Partition    400 MB  Healthy    Hidden
  Volume 4     Z                FAT32  Partition    260 MB  Healthy    System

DISKPART> select volume 4

Volume 4 is the selected volume.

DISKPART> remove letter=z

DiskPart successfully removed the drive letter or mount point.

DISKPART> list volume

  Volume ###  Ltr  Label        Fs     Type        Size     Status     Info
  ----------  ---  -----------  -----  ----------  -------  ---------  --------
  Volume 0     E                       DVD-ROM         0 B  No Media
  Volume 1     C                NTFS   Partition    440 GB  Healthy    Boot
  Volume 2     D   RECOVERY     NTFS   Partition     24 GB  Healthy
  Volume 3         WINRE        NTFS   Partition    400 MB  Healthy    Hidden
* Volume 4                      FAT32  Partition    260 MB  Healthy    System

DISKPART> quit

To be completely honest I was in mostly uncharted waters running these tools to fix my drive issues (Diskpart was entirely new to me).  They simply weren’t tools that I’d had reason to use in my experience to date.  However, as you can see they did fix the file corruption issues that were the source of my failed backups.  I’ve already provided feedback to the WHS group on possibly providing better error messages so users can resolve this issue if they run into it.  However, to be fair to the WHS team, I suspect this issue is a fairly rare issue in that you can tell from the corrupted file names that they were part of a BIOS update which gets stored in a hidden OEM partition on the drive.  My computer still works so the obviously the BIOS update works successfully.  Since I bought the computer with Windows preinstalled use of the OEM partition for the most part is negligible in that the partition exists only for restoring the computer.  Since I’d done that once already in trying to fix this I know that works.  After going through these steps I’m also now backing up my laptop to WHS, too.

Here are few other references on chdkdsk and diskpart you may be interested in:

• Troubleshooting Tools and Strategies
• Chkdsk
• An explanation of CHKDSK and the new /C and /I switches
• A Description of the Diskpart Command-Line Utility
• Assign, change, or remove a drive letter

I hope this was helpful to at least someone.

Thanks.

I recently worked on a case with a customer who needed assistance in configuring the Client Version Filtering for OCS 2007 R2 EE.  For those unfamiliar with this feature here’s a few links that cover the capabilities:

Microsoft Office Communications Server 2007 R2
Configuring Client Version Filtering
https://technet.microsoft.com/en-us/library/dd572488(office.13).aspx

To configure this using the steps in this blog you’ll also need the OCS 2007 R2 Resource Kit Tools installed:

Office Communications Server 2007 R2 Resource Kit Tools
https://www.microsoft.com/downloads/en/details.aspx?familyid=9E79A236-C0DF-4A72-ABA6-9A9602A93ED0&displaylang=en

The easiest way that I‘ve found to set this up is to use a tool in the OCS 2007 R2 Resource Kit called CvcMsiUploader.exe.  In my lab I pulled down the latest public update to Communicator 2007 R2, copied it to the ResKit Folder and then ran this command:

C:\Program Files\Microsoft Office Communications Server 2007 R2\ResKit>CvcMsiUploader.exe /mode:upload /folder:oc /arch:x32 /build:fre /lang:1033 /ee:ocsr2pool.picmepicme.local communicator.msp

To which I received this output:

communicator.msp => \\ocsr2-sql-be\Updates\oc\x32\fre\1033\communicator.msp

Uploaded msi file sucessfully.

In other words, the tool copied the file up to the share and created the folder structure needed.

As a side-note, I’m not sure what to think about the ‘x32’ reference other than it must be a mix-up of ‘x86’ processors supporting 32-bit operating systems.  It’s a minor detail for a Resource Kit Tool that I’m not going to wonder about since it works.

You can check what work the tool did by using this command:

C:\Program Files\Microsoft Office Communications Server 2007 R2\ResKit>CvcMsiUploader.exe /mode:verify /ee:ocsr2pool.picmepicme.local

This should return output similar to this:

C:\Program Files\Microsoft Office Communications Server 2007 R2\ResKit>CvcMsiUploader.exe /mode:verify /ee:ocsr2pool.picmepicme.local

============================================================

Processing Folder: OC

============================================================

Total number of MSI file in store: 1

Validating msi paths...

Info: All msi paths are correct

File: \\ocsr2-sql-be\Updates\OC\x32\fre\1033\communicator.msp

Architecture: x32

Build Type: fre

Language: 1033

----------------------------------------------------------

[ Skipping ]

Info: All msi in store do have same version.

Info: All paths in store contains single installer file

At this point the URL to the update looks like this: 

https://ocsr2pool.picmepicme.local/AutoUpdate/Int/Files/OC/x32/Chk/1033/Communicator.msp

The UNC to the update is:

\\ocsr2-sql-be\Updates\oc\x32\fre\1033\communicator.msp

The client actually receives the updates through IIS so let’s take a look at what this looks like from an IIS perspective; here’s the folder structure:

And here are the authentication settings on that folder:

The last step is to configure OCS so that clients are configured to use the service.  You can access it within the OCS Manager at the Pool Properties:

Forest –> Enterprise Pools –> Pool Name –> Right-Click and Selection Filtering Tools and then Client Version Filter:

Before enabling this functionality you should configure it meet your needs.  I recommend going through this process about quarterly or so since that’s our update cycle on the Communicator client.   Here’s a link to our updates portal if you need it:

Updates Resource Center for Office Communications Server 2007 R2 and Clients
https://technet.microsoft.com/en-us/office/ocs/ee695846.aspx

I’ve had good success in keeping Communicator updated when I configure the filter as follows:

Here’s the text of my configuration:

Major version number:  3

Minor version number:  5

Build number:  6907

Qfe number:  233

Select comparison operation to perform:  <

Select the action to apply to this version:  Block and Upgrade

Folder that contains the upgrade images:  OC

As you’ll see in the user interface there are plenty of options to configure this to meet various needs so my settings above certainly aren’t mandatory; it’s just what works for my lab.  From an IIS perspective you can confirm this is working by looking in the IIS logs where you should see data similar to this:

2011-06-23 13:55:31 192.168.1.45 GET /AutoUpdate/Int/Files/OC/x32/fre/1033/communicator.msp - 443 - 192.168.1.107 Microsoft+BITS/6.7 401 2 5 0

2011-06-23 13:55:31 192.168.1.45 GET /AutoUpdate/Int/Files/OC/x32/fre/1033/communicator.msp - 443 - 192.168.1.107 Microsoft+BITS/6.7 401 1 2148074254 15

2011-06-23 13:55:31 192.168.1.45 GET /AutoUpdate/Int/Files/OC/x32/fre/1033/communicator.msp - 443 PICMEPICME\R2User 192.168.1.107 Microsoft+BITS/6.7 200 0 0 687

Note that the first two GET requests fail because the client tried anonymous authentication first and it fails (we know this by the hyphen denoting an anonymous request and the HTTP 401 response).  However, the authenticated request by PICMEPICME\R2User succeeds (we know this by the HTTP 200 response).

It should be noted that ‘success’ in this context means the download to the client is successful; it doesn’t necessarily mean the client could install the download which likely will require administrative privileges on the machine to complete the installation.

I hope you find this helpful!

Microsoft has added the ability to enhance the Manager/Delegate experience used in Outlook and Exchange scenarios to now include the ability for the delegate to also schedule Office Live Meetings on the manager’s calendar. In this write-up I’ll cover the requirements for this functionality to work and some troubleshooting steps and tools that can be used to ensure the functionality works as expected.

Office Communications Server 2007 R2 and Communicator 2007 R2 Prerequisites:

• At least version 3.5.6907.3 of Office Communicator 2007 R2
• Hotfix 969821 or higher applied to the Office Communications Server 2007 R2 Front-End Server

However, since both of these updates have been superseded by new versions installation of these updates would be preferred:

• Updates Resource Center for Office Communications Server 2007 R2 and Clients
• https://technet.microsoft.com/en-us/office/ocs/ee695846.aspx

Environment Prerequisites

Before manager/delegation of Live Meetings should be expected to work it first needs to be correctly established through Outlook which can be accomplished using the steps in the article below:

• Allow someone else to manage your mail and calendar
• https://office.microsoft.com/en-us/outlook-help/allow-someone-else-to-manage-your-mail-and-calendar-HA010075081.aspx

Please be aware of a limitation in the delegation feature with Office Live Meetings and Communicator:

• 977282  Error message when a delegate user tries to schedule an Office Communications Server 2007 R2 conference on behalf of another user: "You do not have permissions to schedule meetings and conferences on behalf of the owner of this account"
• https://support.microsoft.com/default.aspx?scid=kb;EN-US;977282

Verifying the delegation is in place

When a manager sets a delegate in Outlook what they are really doing is updating an attribute of their Active Directory account called “publicDelegates” which can be viewed with several different tools to confirm it’s in place correct. An easy one to use is ExchDump which you can download from this KB article:

• 839116 Overview of the ExchDump tool for Exchange 2000 Server and for Exchange Server 2003
• https://support.microsoft.com/default.aspx?scid=kb;EN-US;839116

Specifically, run the /user report as in this example:

• Exchdump.exe /user /user:ManagerRepro

After running the command to dump out the manager’s properties open the HTM file that was generated. Expand the “Click for more details” section and look for “publicDelegates” which should be populated with the Distinguished Name of the delegate or delegates of the user:

• publicDelegates : "CN=Delegate Repro,CN=Users,DC=picmepicme,DC=local"

It’s critical to note that without this attribute populated on the manager’s account that no delegate features will work. If this step is missing it must be resolved before continuing.

The next step is to create the client-side registry key (EnableExchangeDelegateSyncUp) that tells Communicator to read the setting in Active Directory and publish it into the OCS database as documented in this KB article:

• 971845 You cannot use a set of features in Office Communicator 2007 R2 that integrate secure delegation for the local Outlook 2007 calendar
• https://support.microsoft.com/default.aspx?scid=kb;EN-US;971845

From an OCS-perspective we can take a look at what happens under the hood on the Manager’s machine when they logon as this is when the database is updated. This happens in the form of SIP SERVICE message to the OCS Database (irrelevant lines have been removed):

From: <sip:ManagerRepro@picmepicme.net>;tag=850e5af551;epid=fbe2655450

To: <sip:ManagerRepro@picmepicme.net>

CSeq: 1 SERVICE

Contact: <sip:ManagerRepro@picmepicme.net;opaque=user:epid:zN6w4yazzluMl5ULEvM4KwAA;gruu>

User-Agent: UCCAPI/3.5.6907.223 OC/3.5.6907.225 (Microsoft Office Communicator 2007 R2)

Content-Type: application/msrtc-setdelegate+xml

<setDelegates xmlns="https://schemas.microsoft.com/2007/09/sip/delegate-management" version="1"><delegate uri="sip:DelegateRepro@picmepicme.net" action="add"/></setDelegates>

To which the server responds with this acknowledgement (irrelevant lines are again removed):

From: "Manager Repro"<sip:ManagerRepro@picmepicme.net>;tag=850e5af551;epid=fbe2655450

To: <sip:ManagerRepro@picmepicme.net>;tag=B4B8A988A6EA05C2452F9A563FE5CFCB

CSeq: 1 SERVICE

<roamingData xmlns="https://schemas.microsoft.com/2006/09/sip/roaming-self" xmlns:cat="https://schemas.microsoft.com/2006/09/sip/categories" xmlns:con="https://schemas.microsoft.com/2006/09/sip/containers" xmlns:sub="https://schemas.microsoft.com/2006/09/sip/presence-subscribers" xmlns:del="https://schemas.microsoft.com/2007/09/sip/delegates">

<delegates xmlns="https://schemas.microsoft.com/2007/09/sip/delegates" version="2">

<delegate uri="DelegateRepro@picmepicme.net" publish="false" redelegate="false"/>

</delegates>

</roamingData>

The final step to check is in the database itself to confirm that the delegation was established correctly; this can be done using the OCS Resource Kit Tool dbanalyze in this manner:

• DBAnalyze.exe /report:user /user:ManagerRepro@picmepicme.net /sqlserver:ocsr2-sql-be

In the output we’re looking for this information toward the bottom of the output:

Delegates

---------

1

Delegate : DelegateRepro@picmepicme.net

If a similar report is run on the delegate user (DBAnalyze.exe /report:user /user:DelegateRepro@picmepicme.net /sqlserver:ocsr2-sql-be) the expected output would have this information at the bottom of the output:

User is a delegate of these local users

---------------------------------------

1

Delegator : ManagerRepro@picmepicme.net

Summary

The OCS delegation feature is used to allow delegates to schedule Live Meetings on behalf of managers. For this feature to work successfully the Outlook delegation by the manager must first be setup. Secondly the client-side registry key must be created so that the necessary SIP traffic will pass to the OCS database and create the required entries. Several tools were discussed in the paper to assist with troubleshooting and ensuring the necessary steps are in place. ]]> https://docs.microsoft.com/archive/blogs/tlyons/resolved-lyncocs-to-windows-live-messenger-public-im-connectivity-pic-service-outage Mon, 21 Feb 2011 13:08:48 GMT https://blogs.technet.microsoft.com/tlyons/2011/02/21/resolved-lyncocs-to-windows-live-messenger-public-im-connectivity-pic-service-outage/ MSN PIC connectivity has been restored. Users may need to sign out and back into their client before seeing successful conversations/presence. Apologies for the inconvenience.

897567    Known issues that occur with public instant messaging and Communications Server
https://support.microsoft.com/default.aspx?scid=kb;EN-US;897567

Barring an unforeseen issues, this is the final notification on this change.

Reference:

https://blogs.technet.com/tlyons/archive/2010/02/26/reminder-additional-windows-live-messenger-pic-federation-ip-address.aspx

897567    Known issues that occur with public instant messaging and Communications Server
https://support.microsoft.com/default.aspx?scid=kb;EN-US;897567

I will provide an additional reminder one week prior to this work going into production.

Reference:

https://blogs.technet.com/tlyons/archive/2010/02/11/postponed-additional-windows-live-messenger-pic-federation-ip-address.aspx

Tomorrow’s planned added capacity has been postponed for technical reasons and to provide additional time for administrators to update their firewall settings if necessary to reflect the additional IP address.  The new date that this work is scheduled to go into place is Friday, March 26, 2010.  For organizations that have chosen to restrict this traffic to specific IP addresses, you must have the list in KB 897567 in place prior to Friday, March 26, 2010 or PIC connectivity with MSN will fail.

897567    Known issues that occur with public instant messaging and Communications Server
https://support.microsoft.com/default.aspx?scid=kb;EN-US;897567

I will provide reminder notifications as the date approaches.

• Front-End servers that need to communicate so the users can IM each other and see presence
• A Front-End server that needs to communicate with an Access Edge Server’s internal interface (remember that an Access Edge Server typically isn’t a member of the domain and therefore may not trust the certificate on the FE as the trust path may only be there for domain members by default).
• An Access Edge Server that needs to communicate with an Access Edge Server in a different OCS deployment that you want to Federate with
• An Access Server that needs to … you get the idea :-)

In this example, I’ll use generic “node 1” and “node 2” names because these steps should be more or less possible in lots of scenarios where two Windows Servers need to talk over MTLS.  Remember that all of this negotiation happens in the context of the computer account (not a service account or user account) so keep that in mind when you open the MMC.  Once you’ve logged onto the server as an Administrator account, try these steps out:

1. Logon to Node 1 and then open the Certificates MMC (remember, computer account not current user)
2. Under the personal store, double-click on the certificate being used for MTLS and then go to the Certification Path tab
3. Click on the issuing certificate and then View Certificate
4. On the Details Tab of that certificate find the serial number and grab the first 4 to 6 digits
5. Now logon to Node 2 and then open the same certificates snap-in (again, computer account)
6. Right-click on Certificates (Local Computer) and select Find Certificates
7. On the Contains tab enter in the 4 to 6 digits you grabbed from Node 1
8. Change the “Look in Field” tab to Serial Number and then click Find Now
9. Review the search results to verify expected behavior of a single certificate being found in the Trusted Root store.  There may be a second certificate is the CA is a Trusted Third-Party
10. You’ll want to repeat this process for each certificate in the certificate chain.  For example, if the MTLS certificate has a chain that is three levels deep (MTLS certificate, intermediate certificate, and root certificate) you’ll want to repeat this process twice:  once for the intermediate certificate and then again for the root certificate.
11. Now repeat this process again in the other direction (from node 2 to node 1) and make changes as needed (i.e., install missing intermediate or root certificates, etc.)
12. If all goes well in terms of the certificate chains, remember that in OCS certificates are read during service startup so you may need to recycle services for any changes to take effect.

NOTE:  If you find any duplicates in your certificate chains a general rule of thumb would be to remove them as it will increase the size of the certificate chain sent down to the remote server during the MTLS negotiation.  Be careful to delete problem certificates from the correct store!  Root certificates should be in the root store; intermediates in the intermediate store.  Some root certificates will also show-up in trusted third-party which I wouldn’t classify as a duplicate.  I’d classify duplicates as more than one instance of the same serial number in the same store or in multiple stores (with the exception being the trusted third-party store).  Also remember that typically if there is an issue with certificate trust, there are multiple ways to getting to the same resolution.  For OCS installations, you’ll very likely find there have been errors logged in the Event Viewer and if SIPStack logging is enabled more errors there as well.  Also, if you’ve successfully logged onto OCS, then the likelihood of a certificate issue is low as that would manifest itself as happening during logon, not after.

I hope these steps are helpful!

Terry