LSASS taking 12 hours to complete at boot up in Windows server 2019 vs 3 minutes in 2012R2

CoderR 1

I have a system with 50K local users (it's create via a script for IIS authentication, this cannot be changed at this time as it's being done by a legacy app).

It's currently running on Windows Server 2012R2 and it takes about 3-4 minutes to boot up - it's been working perfectly for almost a decade now.

I need need to upgrade to Server 2019 or Server 2022. However once the 50K local users are recreated it takes about 12 hours for the server to boot up and login. After analyzing the boot up logs using procmon I noticed that with 2012R2 LSASS is enumerating the SAM users in parallel with other tasks including running LoginUI and once logged in LSASS stop.

Where as in 2019 LSASS has to complete enumerating the SAM users before the login takes place, this takes 12 hours to complete.

Does anyone know how to get 2019 to behave like 2012R2 and have LSASS complete it's SAM validation in parallel with other tasks so it doesn't take 9 hours to boot? Very frustrating the MS broke something that worked great.

CoderR 1 Reputation point

2023-03-31T03:39:20.23+00:00

Looking at the procmon logs for the 2019 server more carefully I noticed that it's not that LSASS is taking a long time to parse the 50K users from SAM, it appears to be stuck in a loop!! LSASS keep reading all the SAM entries over and over again in a loop for 12 hours! Here's an excerpt from the logs showing when it ends one loop and then starts another loop. It's the exact same loop everytime. I can't see the end of the loop because procmon timed out after a few hours of logging but for those few hours it ends the last user, then reads the services NTDS and then restart the SAM user loop again. Anyone may any sense of this?

2:59:42.4202794 PM lsass.exe 680 RegEnumKey HKLM\SAM\SAM\Domains\Account\Users\Names SUCCESS Index: 39,993, Name: X 2:59:42.4202969 PM lsass.exe 680 RegOpenKey HKLM\SAM\SAM\Domains\Account\Users\Names\X SUCCESS Desired Access: Read 2:59:42.4203111 PM lsass.exe 680 RegQueryValue HKLM\SAM\SAM\Domains\Account\Users\Names\X(Default) SUCCESS Type: <Unknown: 82518> 2:59:42.4203232 PM lsass.exe 680 RegCloseKey HKLM\SAM\SAM\Domains\Account\Users\Names\X SUCCESS 2:59:42.4203319 PM lsass.exe 680 RegEnumKey HKLM\SAM\SAM\Domains\Account\Users\Names NO MORE ENTRIES Index: 39,994, Length: 84 2:59:42.4203506 PM lsass.exe 680 RegCloseKey HKLM\SAM\SAM\Domains\Account\Users\Names SUCCESS 2:59:42.4203699 PM lsass.exe 680 RegEnumKey HKLM\SAM\SAM\Domains\Account\Users SUCCESS Index: 9, Name: XX 2:59:42.4203963 PM lsass.exe 680 RegQueryKey HKLM SUCCESS Query: HandleTags, HandleTags: 0x0 2:59:42.4204086 PM lsass.exe 680 RegOpenKey HKLM\SYSTEM\CurrentControlSet\Services\NTDS REPARSE Desired Access: Query Value 2:59:42.4204269 PM lsass.exe 680 RegOpenKey HKLM\System\CurrentControlSet\Services\NTDS SUCCESS Desired Access: Query Value 2:59:42.4204447 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Services\NTDS\DirectoryServiceExtPt NAME NOT FOUND Length: 144 2:59:42.4204594 PM lsass.exe 680 RegCloseKey HKLM\System\CurrentControlSet\Services\NTDS SUCCESS 2:59:42.4204755 PM lsass.exe 680 RegOpenKey HKLM\SAM\SAM\DOMAINS\Account\Users\Names SUCCESS Desired Access: Read 2:59:42.4205331 PM lsass.exe 680 RegEnumKey HKLM\SAM\SAM\Domains\Account\Users\Names SUCCESS Index: 0, Name: XXX 2:59:42.4205462 PM lsass.exe 680 RegOpenKey HKLM\SAM\SAM\Domains\Account\Users\Names\XXX SUCCESS Desired Access: Read 2:59:42.4205604 PM lsass.exe 680 RegQueryValue HKLM\SAM\SAM\Domains\Account\Users\Names\XXX(Default) SUCCESS Type: <Unknown: 42530>
CoderR 1 Reputation point

2023-03-31T03:39:39.83+00:00

Looking at the procmon logs for the 2019 server more carefully I noticed that it's not that LSASS is taking a long time to parse the 50K users from SAM, it appears to be stuck in a loop!! LSASS keep reading all the SAM entries over and over again in a loop for 12 hours! Here's an excerpt from the logs showing when it ends one loop and then starts another loop. It's the exact same loop everytime. I can't see the end of the loop because procmon timed out after a few hours of logging but for those few hours it ends the last user, then reads the services NTDS and then restart the SAM user loop again. Anyone may any sense of this?

2:59:42.4202794 PM lsass.exe 680 RegEnumKey HKLM\SAM\SAM\Domains\Account\Users\Names SUCCESS Index: 39,993, Name: X 2:59:42.4202969 PM lsass.exe 680 RegOpenKey HKLM\SAM\SAM\Domains\Account\Users\Names\X SUCCESS Desired Access: Read 2:59:42.4203111 PM lsass.exe 680 RegQueryValue HKLM\SAM\SAM\Domains\Account\Users\Names\X(Default) SUCCESS Type: <Unknown: 82518> 2:59:42.4203232 PM lsass.exe 680 RegCloseKey HKLM\SAM\SAM\Domains\Account\Users\Names\X SUCCESS 2:59:42.4203319 PM lsass.exe 680 RegEnumKey HKLM\SAM\SAM\Domains\Account\Users\Names NO MORE ENTRIES Index: 39,994, Length: 84 2:59:42.4203506 PM lsass.exe 680 RegCloseKey HKLM\SAM\SAM\Domains\Account\Users\Names SUCCESS 2:59:42.4203699 PM lsass.exe 680 RegEnumKey HKLM\SAM\SAM\Domains\Account\Users SUCCESS Index: 9, Name: XX 2:59:42.4203963 PM lsass.exe 680 RegQueryKey HKLM SUCCESS Query: HandleTags, HandleTags: 0x0 2:59:42.4204086 PM lsass.exe 680 RegOpenKey HKLM\SYSTEM\CurrentControlSet\Services\NTDS REPARSE Desired Access: Query Value 2:59:42.4204269 PM lsass.exe 680 RegOpenKey HKLM\System\CurrentControlSet\Services\NTDS SUCCESS Desired Access: Query Value 2:59:42.4204447 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Services\NTDS\DirectoryServiceExtPt NAME NOT FOUND Length: 144 2:59:42.4204594 PM lsass.exe 680 RegCloseKey HKLM\System\CurrentControlSet\Services\NTDS SUCCESS 2:59:42.4204755 PM lsass.exe 680 RegOpenKey HKLM\SAM\SAM\DOMAINS\Account\Users\Names SUCCESS Desired Access: Read 2:59:42.4205331 PM lsass.exe 680 RegEnumKey HKLM\SAM\SAM\Domains\Account\Users\Names SUCCESS Index: 0, Name: XXX 2:59:42.4205462 PM lsass.exe 680 RegOpenKey HKLM\SAM\SAM\Domains\Account\Users\Names\XXX SUCCESS Desired Access: Read 2:59:42.4205604 PM lsass.exe 680 RegQueryValue HKLM\SAM\SAM\Domains\Account\Users\Names\XXX(Default) SUCCESS Type: <Unknown: 42530>

1 answer

Limitless Technology 43,926 Reputation points

2023-03-31T10:31:44.8066667+00:00
Hello

Thank you for your question and reaching out. I can understand you are having query\issues related to LSASS taking huge time.

Please check and scan for Check for malware and also perform Full scan with your Antivirus on 2012 R2 server.

Disable any Local Windows firewall for temporary purpose.

Perform scandisk using below command as some due to disk issues it may tame take time to boot up.

chkdsk /f /r c:\

Please also consider to Migrate FSMO role to new Additional DC having 2019 or 2022 installed.

--If the reply is helpful, please Upvote and Accept as answer--
Please sign in to rate this answer.

0 comments No comments
Sign in to comment