question

Researcher-8306 avatar image
0 Votes"
Researcher-8306 asked Researcher-8306 commented

Lsass.exe consuming high CPU usage in web server with ASP .NET applications

We have two web servers hosting ASP .NET applications on Windows Server 2012 R2 machine with IIS version 8.5, which is not a Domain Controller. (The server is hosted with AWS). The server has 3 ASP.Net domains and one Wordpress domain.
The CPU and RAM configuration are:
Server 1: 8 GB RAM, 2 vCPU, 2.40 Ghz (E5-2676 v4)
Server 2: 4 GB RAM , 2 vCPU, 2.30 Ghz (E5-2686 V4)

We are observing very high CPU usage consumed by the LSASS.exe sometimes, which is the Local Security Authority process, with overall CPU usage touching 100% in some cases.
We checked by stopping the IIS service, during which the overall CPU usage came down significantly, below 10%. But when each IIS site was turned on, the CPU usage started going up high again. We have identified that the issue is not related to Active Directory as the AD role is not enabled. The Lsass.exe version in both the servers is 6.3.9600.17415. We are currently in the process of updating lsass.exe/lsasrv.dll to a higher version through KB update.

We have tried stopping the domains one by one and enabling again. We could not point out exactly for which domain the CPU usage is rising. We randomly enabled few websites and checked, the CPU usage wasn’t specific to any particular domain.
The number of HTTPS connection entries as per IIS logs are:

Server 1:
normal traffic: around 1000 requests per minute.
traffic when lsass.exe CPU was high: 4500+ requests per minute.

Server 2:
normal traffic: around 500-1000 requests per minute.
traffic during lsass.exe CPU was high: 2,200+ requests per minute.


In the event logs, we found the following: Event IDs – 4625 (external users attempting to login using invalid credentials), 4797 related to account security (checking for blank passwords). There are also Event IDs- 36874 and 36888 in System Log related to SChannel and TLS errors. However, these events were also logged on days when the lsass.exe CPU usage was normal. We have checked the IIS, FTP, HTTPERR and antivirus logs and have not found anything unusual. The SQL log entries have a few of "Error: 18056, Severity: 20, State: 29", which says the client is unable to reuse a session with SPID <xx> which have been logged before the Lsass.exe CPU usage started going high.

We are using Symantec Endpoint Protection with the following settings/features enabled: Virus and spyware protection, proactive threat protection, Network and Host exploit mitigation, File system AutoProtect, Download Insight, Early-launch anti-malware among others, including SONAR. Symantec Firewall, Smart DHCP, WINS, DNS, Port Scan Detection, Allow only application traffic, Active Response Settings (automatically blocks an attacker's IP address: 600 seconds). Also Network Intrusion Prevention, Memory Exploit Mitigation. The Windows Firewall is also ON in both the host servers.
We are using Anonymous authentication, with the impersonation settings in IIS and ASP .NET shown as below:

227616-s1.jpg

227721-s2.jpg


The basic DDOS protection in the AWS Lightsail is enabled. The host Windows Server 2012 R2 in
Server 1 is having security updates up to KB2896496 installed (in 2018).
Server 2 is having security updates up to KB4338331 installed (in 2018).
We checked the below KB article from Microsoft, but since we do not have AD role enabled, the Active Directory diagnostic logging feature is not active.


https://docs.microsoft.com/en-us/troubleshoot/windows-server/identity/troubleshoot-high-lsass.exe-cpu-utilization

We renewed the SSL certificates of both the servers around June and July this year and have confirmed that the key-size of the renewed certificates is the same as previous.

Could someone assist us in knowing why the lsass.exe is consuming unusually high CPU usage? Is there anything further we might need to check other than update the lsass.exe version?
(Please feel free to request any further details/specification/server data if required).

EDIT 04-08-2022: Updated SSL certificate changes.

EDIT 05-08-2022:
We analyzed the dump of lsass.exe using the Debug Diagnostic tool and the information was :
"The following threads in lsass.dmp are processing an inbound RPC call over LRPC to the lsasspirpc endpoint."
Each of the threads have below stack and when searched online we did not find specific information. Could you please let us know what these indicate, in the context of lsass.exe consuming high CPU?

 *ntdll!NtWaitForWorkViaWorkerFactory+a      
 ntdll!TppWorkerThread+746      
 kernel32!BaseThreadInitThunk+22      
 ntdll!RtlUserThreadStart+34



EDIT 01-09-2022:

One solution that we want to do right now is to increase the Servers instance specs to compensate the high CPU usage required by lsass.exe

What we have tried:

(For workload on Jmeter with values as users : 1000 , ramp up period : 10 , loop : 1)

  1. We have tested the Https aspx site hosted in IIS 8.5 on Windows Server 2012 R2 using Jmeter tool. The lsass.exe takes about 26%.

  2. We have tested the same Https aspx site hosted in IIS 10 on Windows Server 2019 using Jmeter tool. The lsass.exe results are the same. ( Same instance hardware specs)

  3. Tried Https site hosted in apache. Lsass.exe is not used.


But we are not able to determine the CPU sizing required to handle the HTTPS requests. Because this 26% of usage can be lowered when using a Higher capacity CPU , how to determine which one. Can you guys help with this .

  1. Is it normal CPU value that lsass.exe is consuming in our case for the tested Workload?

  2. Is there any way to calculate the CPU usage in percent for any CPU on per app basis before hand?







windows-server-iiswindows-server-2012dotnet-aspnet-general
s2.jpg (23.1 KiB)
s1.jpg (25.7 KiB)
· 6
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @Researcher-8306,

As lsass.exe works on several part of HTTPS sessions, it is not trivial to tell what exactly happens without things like dump analysis. Possible causes can be related to the certificate you use (some certificates require more processing than others), but that's only one of the many possibilities. If you have unknowingly replaced a 2048 SSL certificate with a 4096 certification, there are chances you are facing lsass.exe high CPU usage. Hence, the best bet is just to switch to the original certificate where everything was usually working.

0 Votes 0 ·

Hello YurongDai-MSFT, we had renewed the server SSL certificates around a month ago, but we have confirmed that there are no changes in the key-sizes in the renewed certificates.

0 Votes 0 ·

Hi @@Researcher-8306,
See which module/function is causing the cpu usage in lsass,try to analyze CPU usage with Windows Performance Recorder/Analyzer.

0 Votes 0 ·
Show more comments

0 Answers