Replication issues after ungraceful DC restore/restore from backup

Question

Hi,

I had a major issue with one of my domain controllers where it could not be gracefully demoted and had to be restored from backup. I know this is a no no but there was no other option at the time. Unfortunately I went back too far, 1 month to be precise, and since then my domain has had some big replication issues. I have been using dcdiag to try and diagnose the issues and I am receiving this error when I attempt to replicate to any of the other DCs from my FSMO master:

TEST: Authentication (Auth)
Error: Authentication failed with specified credentials
[Error details: 1326 (Type: Win32 - Description: The user name or password is incorrect.) - Add connection failed]

           TEST: Basic (Basc)  
              Error: No LDAP connectivity  
              Error: No WMI connectivity  
              [Error details: 0x80070005 (Type: HRESULT - Facility: Win32, Description: Access is denied.) - Connection to WMI server failed]  
              No host records (A or AAAA) were found for this DC  

I do see host records for all of the DCs in ADS&S so I don't understand that error message. At first I believed that this had to do with KDC/Kerberos more than anything because the secure channel between my failed DC & the rest of the domain was broken. Trying to fix the secure channel has been a headache, not really sure where to go from here.

I did find this article useful and I think it pertains to me: https://support.microsoft.com/en-us/help/2002013/active-directory-replication-error-5-access-is-denied

These are the resources/guides that I have tried using:

• https://learn.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2003/cc780728(v=ws.10)?redirectedfrom=MSDN
• https://kb.eventtracker.com/evtpass/evtPages/EventId_4_Microsoft-Windows-Security-Kerberos_63365.asp#:~:text=This%20indicates%20that%20the%20password,the%20client%20realm%20(%254).
• https://social.technet.microsoft.com/Forums/windowsserver/en-US/e9c162cb-1e26-43e0-80df-73c491c22aac/replication-error-quotthe-target-principal-name-is-incorrectquot?forum=winserverDS
• https://social.technet.microsoft.com/Forums/ie/en-US/f8a93cde-f1de-47b6-b85a-781c795825f7/kerberos-event-id-4-krbaperrmodified?forum=winserverDS
• https://support.microsoft.com/en-us/help/875495/how-to-detect-and-recover-from-a-usn-rollback-in-windows-server-dc
• https://social.technet.microsoft.com/Forums/lync/en-US/70e5f4e3-ca05-4868-a217-bcd7c6fcfdc4/active-directory-not-replicating-after-full-server-restore?forum=winserverDS
• https://support.microsoft.com/en-us/help/2022387/active-directory-replication-error-8453-replication-access-was-denied
• https://social.technet.microsoft.com/Forums/lync/en-US/d08ff762-18d0-4237-a7f6-5a2b3084999f/error-8341-and-58-in-repadmin-replsummary?forum=winserverDS

Any leads would be appreciated as I'm really trying everything to repair this. Once I figure out one error, it leads to another, and so on... Thank you

Also, I did try posting this in TechNet and it keeps redirecting me to here... please let me know if this is incorrect.

Answer 1

Restoring a domain controller in a multi DC environment is not recommended. The much safer / cleaner option is to seize roles to a healthy one (if needed)
https://support.microsoft.com/en-us/help/255504/using-ntdsutil-exe-to-transfer-or-seize-fsmo-roles-to-a-domain-control

then perform cleanup.
https://learn.microsoft.com/en-us/windows-server/identity/ad-ds/deploy/ad-ds-metadata-cleanup

then rebuild the failed one from scratch by using dcdiag / repadmin tools to verify health correcting all errors found before starting. Then stand up the new replacement, patch it fully, license it, join existing domain, add active directory domain services, promote it also making it a GC (recommended), transfer FSMO roles over (optional), transfer pdc emulator role (optional), use dcdiag / repadmin tools to again verify health.

--please don't forget to Accept as answer if the reply is helpful--

Answer 2

Hello,

Thank you so much for posting here.

According to our description, what restore did we preform, authoritative restore or nonauthoritative restore? Besides, how about our other DCs? Do they work properly? It is suggested that we could backup the healthy DC before any operations.

If there are so many issues with this DC, as Dave mentioned, we could forcefully demote the DC, and done a meta data cleanup. Then promote it as a new DC.

Hope the information is helpful. For any question, please feel free to contact us.

Best regards,
Hannah Xiong

Answer 3

Why would restoring a DC from backup that is a week old not break replication, but going back a month broke everything?

Neither is recommended. The much safer / cleaner option is to seize roles to a healthy one (if needed)
https://support.microsoft.com/en-us/help/255504/using-ntdsutil-exe-to-transfer-or-seize-fsmo-roles-to-a-domain-control

then perform cleanup.
https://learn.microsoft.com/en-us/windows-server/identity/ad-ds/deploy/ad-ds-metadata-cleanup

then rebuild the failed one from scratch by using dcdiag / repadmin tools to verify health correcting all errors found before starting. Then stand up the new replacement, patch it fully, license it, join existing domain, add active directory domain services, promote it also making it a GC (recommended), transfer FSMO roles over (optional), transfer pdc emulator role (optional), use dcdiag / repadmin tools to again verify health.

--please don't forget to Accept as answer if the reply is helpful--

Answer 4

What happens when you try? If the failed one happens to be the fsmo role holder then you can seize the roles to a healthy one.
https://support.microsoft.com/en-us/help/255504/using-ntdsutil-exe-to-transfer-or-seize-fsmo-roles-to-a-domain-control

--please don't forget to Accept as answer if the reply is helpful--

Answer 5

Hi Alex,

Thank you so much for your feedback.

May I know the current situation of our issue? As you mentioned, the FSMO roles could not be transferred. How about seizing the roles to a healthy DC?

As per my understanding, the FSMO master roles will not replicate among the DCs. You are worried about transferring the FSMO roles to another DC as the replication is not working properly. But to transfer the FSMO roles or seize the FSMO roles will make sure that our whole AD environment will always work properly. About when to transfer or seize FSMO roles, we could refer to:
https://support.microsoft.com/en-us/help/255504/using-ntdsutil-exe-to-transfer-or-seize-fsmo-roles-to-a-domain-control

After transferring or seizing the FSMO roles, we could run the command "netdom query FSMO" on all the DCs to verify the DC holding FSMO roles.

We understand that we would like to fix the replication between the FSMO DC and the other 8 DCs, not to demote the FSMO DC. If so, since we preformed the restore from backup, we need to figure out whether the replication failure is caused by USN rollback. If it is caused by USN rollback, it is suggested that we would remove the DC from domain. For more information about this, we could refer to:
https://support.microsoft.com/en-us/help/875495/how-to-detect-and-recover-from-a-usn-rollback-in-windows-server-dc

If there is no USN rollback, we will go next and try to fix the replication. To fix the replication issue, we need to figure out all the replication errors first. To check about this, we could run the below commands:

repadmin /showrepl * /csv >C:\showrepl.csv (run the command on one of the DCs)
repadmin /showrepl >c:\showrepl.txt (run the commands on all the DCs)
repadmin /replsum >c:\replsum.txt (run the commands on all the DCs)

For any question, please feel free to contact us.

Best regards,
Hannah Xiong