Windows 2019 hangs on 50,000 users, 2012R2 works just fine - what's going on?

Question

I've been using Windows server 2012R2 Standard for almost a decade now. I have apps running on the server which have created about 50,000 local users. These are local users created using WMIC scripts and eq. API's. The purpose is to allow for easy integration with IIS authentication. The app creates a local user account and IIS uses the built in Windows authentication to authenticate the user.

It's been running great on 2021R2 but I recently decided to upgrade to a 2019 Datacenter edition so I created a new 2019 server and used the same scripts to recreate the 50,000 users on the 2019 server.

However when I reboot the 2019 server it just hangs for hours at the spinning circle and takes up 100% CPU. It never get the login screen. I've never had this issues with 2012R2, to test it I spun up  a new 2012R2 server and recreated the same 50,000 local users with the scripts and it took a few minutes to reboot but that was it.

Is there something different about 2019 server datacenter vs 2012r2 standard which causes the 2019 server to hang on boot when there are 50,000 local users? Is it trying to process all the users before presenting the login (I've set it to login automatically to an admin account but it never seems to get there). I'm at my wits end now and I'm open to suggestions on how to debug this further or what settings should I look at? I've checked out the group security and logon policies and they're both the same for the 2012 and 2019 servers. What am I overlooking?

They're both running on AWS with 1GB RAM. Thanks in advance and please don't hesitate to throw out whatever ideas you may have. I cannot change the architecture at this time and the easiest way to integrate IIS with authentication is to create a local user account. They don't need to have local login privileges only network login (for IIS), so these accounts are added to a custom Group on the windows server and removed from the standard Users group.

Answer 1

Hello there,

Have you checked the Hardware requirements for Windows Server 2019 ?

This article outlines the minimum hardware requirements to run Windows Server. If your computer has less than the minimum requirements, you will not be able to install this product correctly. Actual requirements will vary based on your system configuration and the applications and features you install. https://learn.microsoft.com/en-us/windows-server/get-started/hardware-requirements

https://learn.microsoft.com/en-us/windows-server/get-started/hardware-requirements

The below thread discusses the same issue and you can try out some troubleshooting steps from this and see if that helps you to sort the Issue.

https://learn.microsoft.com/en-us/answers/questions/853305/windows-server-2019-std-spinning-dots-on-load

Hope this resolves your Query !!

--If the reply is helpful, please Upvote and Accept it as an answer–

Answer 2

@Limitless Technology thanks for your suggestions. I checked the minimum requirements and it's fine, infact these configurations are recommended by AWS Lightsail when setting up a server and the 2021R2 has been running for almost a decade now without any issues.

Just for kicks, I tried a new AWS 2019 server with 4 and 8GB RAM and also more processors, same result. The scripts runs, add 40K users and everything is working fine up and until I restart the machines and then I end up with the spinning dots (on AWS I can't see it because RDP isn't up or isn't responding) but when tested with a local VM I can see it there, so it's no the size of the RAM/hardware which appears to be the issue here.

I also looked at the Bypass Traverse policy settings and it's set correctly the defaults. I checked the settings on the 2012R2 server it's identical to the 2019 server (both have Everyone listed).

I appreciate the tips - can you think of anything else? Is there to export all the configuration/settings of a windows server? That way I can export the 2021R2 and do a diff against the 2019 and see what settings are different.

Answer 3

@Limitless Technology I have one more insight, I created a new 2019 server, added the 50k new account but then just left the machine without rebooting. I logged out of the RDP session, wait for a few hours and tried to RDP back in and now it's hung on "Please wait for Local Session Manager" and it just times out, the console shows the CPU usage spikes to 100%. So it seems like something with Local Session Manager is causing the issue. I did the same with a 2022 server also and the same results (so it's not specific to 2019). Any ideas?

Answer 4

Okay so the server with 50K users takes about 12 hours to complete the boot up and login process.

I'm almost convinced it's a SAM issue. When I look at the server system event logs (without any users created, clean install), the Directory-Services-SAM events, there are 3 SAM event logs that complete within about 1 second. On the server with the 50k users, those same 3 events are 12 hours apart, the first one right after Wininit and the the next one 12 hours later (when login takes place)

So the first one after Wininit:

Remote calls to the SAM database are being restricted using the default security descriptor: O:SYG:SYD:(A;;RC;;;BA).

For more information please see http://go.microsoft.com/fwlink/?LinkId=787651.

The second one after 12 hours:

The domain is configured with the following minimum password length-related settings.

MinimumPasswordLength: 0

MinimumPasswordLengthAudit: -1

For more information see https://go.microsoft.com/fwlink/?LinkId=2097191.

And the third one immediately thereafter:

The security account manager is now logging periodic summary events for remote clients that call legacy password change or set RPC methods.

For more information please see https://go.microsoft.com/fwlink/?linkid=2150956.

Any thoughts on how to get SAM to speed up or not load all profiles at boot up?

Answer 5

Looking at the procmon logs for the 2019 server more carefully I noticed that it's not that LSASS is taking a long time to parse the 50K users from SAM, it appears to be stuck in a loop!! LSASS keep reading all the SAM entries over and over again in a loop for 12 hours! Here's an excerpt from the logs showing when it ends one loop and then starts another loop. It's the exact same loop everytime. I can't see the end of the loop because procmon timed out after a few hours of logging but for those few hours it ends the last user, then reads the services NTDS and then restart the SAM user loop again. Anyone may any sense of this?

2:59:42.4202794 PM    lsass.exe    680    RegEnumKey    HKLM\SAM\SAM\Domains\Account\Users\Names    SUCCESS    Index: 39,993, Name: X 2:59:42.4202969 PM    lsass.exe    680    RegOpenKey    HKLM\SAM\SAM\Domains\Account\Users\Names\X    SUCCESS    Desired Access: Read 2:59:42.4203111 PM    lsass.exe    680    RegQueryValue    HKLM\SAM\SAM\Domains\Account\Users\Names\X(Default)    SUCCESS    Type: <Unknown: 82518> 2:59:42.4203232 PM    lsass.exe    680    RegCloseKey    HKLM\SAM\SAM\Domains\Account\Users\Names\X    SUCCESS     2:59:42.4203319 PM    lsass.exe    680    RegEnumKey    HKLM\SAM\SAM\Domains\Account\Users\Names    NO MORE ENTRIES    Index: 39,994, Length: 84 2:59:42.4203506 PM    lsass.exe    680    RegCloseKey    HKLM\SAM\SAM\Domains\Account\Users\Names    SUCCESS     2:59:42.4203699 PM    lsass.exe    680    RegEnumKey    HKLM\SAM\SAM\Domains\Account\Users    SUCCESS    Index: 9, Name: XX 2:59:42.4203963 PM    lsass.exe    680    RegQueryKey    HKLM    SUCCESS    Query: HandleTags, HandleTags: 0x0 2:59:42.4204086 PM    lsass.exe    680    RegOpenKey    HKLM\SYSTEM\CurrentControlSet\Services\NTDS    REPARSE    Desired Access: Query Value 2:59:42.4204269 PM    lsass.exe    680    RegOpenKey    HKLM\System\CurrentControlSet\Services\NTDS    SUCCESS    Desired Access: Query Value 2:59:42.4204447 PM    lsass.exe    680    RegQueryValue    HKLM\System\CurrentControlSet\Services\NTDS\DirectoryServiceExtPt    NAME NOT FOUND    Length: 144 2:59:42.4204594 PM    lsass.exe    680    RegCloseKey    HKLM\System\CurrentControlSet\Services\NTDS    SUCCESS     2:59:42.4204755 PM    lsass.exe    680    RegOpenKey    HKLM\SAM\SAM\DOMAINS\Account\Users\Names    SUCCESS    Desired Access: Read 2:59:42.4205331 PM    lsass.exe    680    RegEnumKey    HKLM\SAM\SAM\Domains\Account\Users\Names    SUCCESS    Index: 0, Name: XXX 2:59:42.4205462 PM    lsass.exe    680    RegOpenKey    HKLM\SAM\SAM\Domains\Account\Users\Names\XXX    SUCCESS    Desired Access: Read 2:59:42.4205604 PM    lsass.exe    680    RegQueryValue    HKLM\SAM\SAM\Domains\Account\Users\Names\XXX(Default)    SUCCESS    Type: <Unknown: 42530>