Active Directory Issues

Question

Current Domain Environment:
2 Domain Controllers running Server 2016 Standard. Both machines are hosted in VMWare on premise. Both machines have Active Directory Domain Services and DNS Server installed. One server is also running DHCP Server.
1 member server running Server 2019 Datacenter hosting Active Directory Certificate Services and Network Policy Server using Radius as part of MFA for VPN users.
GPO pushes certificates to computers in VPN group
1 member server running Server 2019 Datacenter hosting Microsoft Exchange 2019.
Several Server 2016 and Server 2019 servers hosting various applications and services.
Domain Functional Level 2016

If either of the 2016 Domain Controller servers is restarted for ANY reason, it is usually not possible to log in to the server and Domain services and DHCP do not function. When the password for the user account is entered and the “submit” button or enter key is pressed, nothing happens. The cursor is either sent to the beginning of the password field or to the beginning of the username field. Changing users or retrying the logon does not change the behavior. The machine must be restarted several times before finally being able to logon.
We tried adding 2 new domain controllers using Server 2019 Datacenter. Once it they were promoted to a domain controller, the console could not be logged onto. It would give the message or “Bad Username or Password”. If they was left running for a period of time and a computer on the network tried to authenticate a user account against this domain controller, the account would be denied with “Bad Username or Password”. The server could be connected to and managed with Server Manager or PowerShell. Remote Desktop sessions could not connect and were give the “Bad Username or Password” message. Server 2019 Domain controller was powered off to prevent valid user from being denied.

Running the AD Replication Status Tool, one of the 2019 DCs gives 2 errors for 2 of the other DCs. Replication Error 1256 and 1722. Running DCDIAG from one of the 2016 DCs, all tests pass except it shows "A recent replication atempt failed:" and it lists the 2 errors: 1256 & 1722

Obviously, something is broken and I have exhausted myself trying to track this down.

Answer 1

That article is for Server 2012 but I searched for the same thing with Server 2016 and went through the steps. Rebooted DC1 and still had to restart multiple times before I could log in.

Even if I shut down DC1 and then reboot AD02, I still have problems logging in. I still have to reboot numerous times before it will finally log in. It seems like there is more than DFS Replication problems.

Answer 2

It seems like there is more than DFS Replication problems.

That may be true. I'd check the event logs for errors since last boot.

Answer 3

IsmServ service terminated with the following error:

What other roles are installed?

Dfs service

Did the service start? are there some namespaces here?

Windows Defender Service service terminated

What operating system? Firewall needs to be running

Another option if you believe AD02 is a lost cause AND DC1 is healthy is to seize roles to DC1,
https://learn.microsoft.com/en-us/troubleshoot/windows-server/identity/transfer-or-seize-fsmo-roles-in-ad-ds

do cleanup
https://learn.microsoft.com/en-us/windows-server/identity/ad-ds/deploy/ad-ds-metadata-cleanup
https://techcommunity.microsoft.com/t5/itops-talk-blog/step-by-step-manually-removing-a-domain-controller-server/ba-p/280564

then rebuild the other from scratch. If you're not sure or comfortable with this then start a case here with product support.
https://support.serviceshub.microsoft.com/supportforbusiness

--please don't forget to upvote and Accept as answer if the reply is helpful--

Answer 4

Other Roles: AD DS, DHCP, DNS

DFS Service, Hard to tell, had to reboot again but eventually, yes.

OS: Windows Server 2016 Datacenter - Firewall was disabled during troubleshooting at some point

We already tried all of that. DC1 is a fresh build that was promoted to DC and immediately began exhibiting the same problems. I am leery to removed AD02 but not sure it makes a difference now. We are discussing a call with MS as an option at this point.

IsmServ service terminated with the following error:  

What other roles are installed?

Dfs service  

Did the service start? are there some namespaces here?

Windows Defender Service service terminated  

What operating system? Firewall needs to be running

Another option if you believe AD02 is a lost cause AND DC1 is healthy is to seize roles to DC1,
https://learn.microsoft.com/en-us/troubleshoot/windows-server/identity/transfer-or-seize-fsmo-roles-in-ad-ds

do cleanup
https://learn.microsoft.com/en-us/windows-server/identity/ad-ds/deploy/ad-ds-metadata-cleanup
https://techcommunity.microsoft.com/t5/itops-talk-blog/step-by-step-manually-removing-a-domain-controller-server/ba-p/280564

then rebuild the other from scratch. If you're not sure or comfortable with this then start a case here with product support.
https://support.serviceshub.microsoft.com/supportforbusiness

Answer 5

From the application log:

Event 8193 VSS Unexpected error calling routine RegOpenKeyExW(-2147483646,SYSTEM\CurrentControlSet\Services\VSS\Diag,...). hr = 0x80070005, Access is denied.

MSDTC 4427 Failed to initialize the needed name objects. Error Specifics: hr = 0x800706d3, com\complus\dtc\dtc\msdtcprx\src\dtcinit.cpp:575, CmdLine: C:\WINDOWS\system32\dllhost.exe /Processid:{02D4B3F1-FD88-11D1-960D-00805FC79235}, Pid: 2556

Complus 4691 (Related to MSDTC) The run-time environment was unable to initialize for transactions required to support transactional components. Make sure that MS-DTC is running. (DtcGetTransactionManagerEx(): hr = 0x8004d027)

Application Error 1000

Faulting application name: lsass.exe, version: 10.0.14393.1770, time stamp: 0x59bf2fb2
Faulting module name: msvcrt.dll, version: 7.0.14393.0, time stamp: 0x57899b47
Exception code: 0xc0000005
Fault offset: 0x0000000000073f5a
Faulting process id: 0x210
Faulting application start time: 0x01d7b4979fc62a94
Faulting application path: C:\WINDOWS\system32\lsass.exe
Faulting module path: C:\WINDOWS\System32\msvcrt.dll
Report Id: b4ad9eaa-39b3-4d9b-982a-439aa833fea8
Faulting package full name:
Faulting package-relative application ID:

Event 1015 Wininit (related to lsass.exe) A critical system process, C:\WINDOWS\system32\lsass.exe, failed with status code c0000005. The machine must now be restarted. (it actually restarted itself)

From the System Log

Event 10119 WinRM
The WinRM service is unable to start because of a failure during initialization.

Additional Data
The error code is 1115.

Event 5000 LsaSrv
The security package NTLM generated an exception. The exception information is the data.

Event 7023 Service Control Manager
The Spooler service terminated with the following error:
%%2147944147

Event 7001 Service Control Manager
The Dfs service depends on the RemoteRegistry service which failed to start because of the following error:
The operation completed successfully.

Event 15016 HttpEvent
Unable to initialize the security package WDigest for server side authentication. The data field contains the error number.

Event 15016 HttpEvent
Unable to initialize the security package NTLM for server side authentication. The data field contains the error number.

Event 15016 HttpEvent
Unable to initialize the security package Negotiate for server side authentication. The data field contains the error number.

Event 15016 HttpEvent
Unable to initialize the security package Kerberos for server side authentication. The data field contains the error number.

Event 15016 HttpEvent
Unable to initialize the security package Basic for server side authentication. The data field contains the error number.

Event 7023 Service Control Manager
The IsmServ service terminated with the following error:
A directory service error has occurred.

Event 7023 Service Control Manager
The Windows Defender Service service terminated with the following error:
%%2147550931

Event 1006 Microsoft-Windows-DHCP-Server
The DHCP service failed to start as a RPC server. The following error occurred :
The authentication service is unknown.

Event 1008 Microsoft-Windows-DHCP-Server
The DHCP service is shutting down due to the following error:
The authentication service is unknown.

Event 7023 Service Control Manager
The DHCP Server service terminated with the following error:
The authentication service is unknown.

Event 1052 GroupPolicy (Microsoft-Windows-Group-Policy)
The processing of Group Policy failed. Windows could not determine the role of this computer. Role information (Workgroup, Member Server, or Domain Controller) is required to process Group Policy.