question

niklas-sjogren avatar image
0 Votes"
niklas-sjogren asked dstaulcu commented

Sysmon 13.2x using huge amount of handles on servers?

I’m getting reports from our server guys that the later versions of Sysmon (13.2x) is slowly eating up all handles in the systems…
Seems to affect all versions och windows server

Image shows example in order:
Win 2019 server
Win 2016 server
Win 2012 R2 server
129689-img1.jpg


15 600 000 on a 2016 server (!?)

129699-img2.jpg

Is this anything anyone else have noticed?
Possible bug in Sysmon ?



windows-sysinternals-sysmon
img1.jpg (65.6 KiB)
img2.jpg (34.8 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

niklas-sjogren avatar image
0 Votes"
niklas-sjogren answered

Adding som more information in this case...

I see this problem on all systems (servers and clients) running v.13.22 - v.13.24...
My PC climbs during the workday to 60.000-70.000 handles, and systems not restarted (like servers) even more...

After some testing i found that version 13.20 works without this problem, so I am now downgrading to this version...

No one else seeing this???

Try:
while ($true) {$foo=get-date -Format "HH:mm:ss";write-host "$foo Handles: " -NoNewline;Get-Process -Name sysmon64 | select -ExpandProperty Handles; sleep -Seconds 60;$foo=$null}

during the day on your computer...

Let me know if you see the same problem...

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

dstaulcu avatar image
0 Votes"
dstaulcu answered niklas-sjogren commented

Have you deduced which event type is causing the leak? Id start by disabling file delete logging...

· 4
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

I have done tests with and without Event 26, but with the same results when running version 13.24.
We see this on all windows versions (win10, 2012R2, 2016 and 2019)

Handles slowly climbing until process or server is restarted.


0 Votes 0 ·

Tested removing on event at a time, and I think that EVT 7 is the problem in my case....

the full config works fine in version 13.20

0 Votes 0 ·

Thanks for going through the process of elimination. With the component narrowed down, I recommend sharing your observations directly with the Sysinternals Site Support and Bug Reporting <syssite@microsoft.com> email group.

1 Vote 1 ·

Adding info from ProcessExplorer...

136548-sysmonhandels.jpg


0 Votes 0 ·
sysmonhandels.jpg (889.8 KiB)
niklas-sjogren avatar image
0 Votes"
niklas-sjogren answered niklas-sjogren commented

I today tested the 13.30 version to see if my "bug" was corrected..
But sadly I have to say that CPU handles continue to grow during the day even with this version...

If I remove event 7 from my configuration it works just fine, but no logging of Event 7..

we are collecting all Image Loaded events and have tested the following configuration for this:
<ImageLoaded condition="more than">0</ImageLoaded> - works, but CPU handles climb fast...
<ImageLoaded condition="contains">\</ImageLoaded> - works, but CPU handles climbs slowly (to 11.000+ during my workday)

Both of these configurations work fine in version 13.20 but not in any of the following versions of Sysmon...

· 3
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Did you ever send an email to syssite@microsoft.com and receive engagement on the issue?

0 Votes 0 ·

yes! I did and got no reply or comment...

0 Votes 0 ·

I emailed sysite@microsoft.com once again adding my comment above regarding the latest version...

0 Votes 0 ·
niklas-sjogren avatar image
0 Votes"
niklas-sjogren answered dstaulcu commented

An update in this case...(If anyone still is reading...)
No! I have not yet received any reply or comment from sysite@microsoft.com :-/ (I am guessing no one can recreate this..)

We collect all Evt 7 in the config we are running by "<ImageLoaded condition="contains">\</ImageLoaded>"

Did some more testing and found the following:

  1. If I remove all "Exclude" rules in Evt 7, everything works (but I collect a massive amount of Evt7) no increase in handles

  2. If I remove all "Exclude" rules for McAfee, collect works but I still get a slow increase in handles during the day.

  3. installed sysmon 13.30 and our config on a clean windows installation (Defender as AV), collect works but I get a slow increase in handles during the day.

  4. Took all of Olaf Hartongs configs for collecting Evt7, selected items are collected and no increase in handles...

My guess is that collectin all and adding many exclude rules for Evt7 are creating this problem, but this worked fine up till version 13.20...

I am now slowly adding items to Hartongs config to see how far I can go...



· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Thanks for the update. I've been holding off on upgrade of 13.30 pending clarity on the issue you reported.
This sounds like the sort of issue where clues for root cause would be discovered through evaluation of a memory dump. Do you work in the type of organization where sharing a memory dump is feasible? --I imagine that is what the developers would ask if engagement were to start and others can not readily reproduce what you are experiencing.
If you are unable to share memory dumps, I'd imagine your process of elimination in Olaf configs would also be useful to further narrow the problem source



0 Votes 0 ·
foxmsft avatar image
0 Votes"
foxmsft answered dstaulcu commented

Indeed, can't seem to get a repro on this one. However, managed to make some improvements regarding token cleanup. I don't think your emails got through to that mailbox, would you be willing to test an experimental Sysmon build? Just drop me an email on this account at hotmail and I'll reply from my Microsoft account.

· 7
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi!
did a quick test with 13.31
seems to be worse than the previous version :-/ at least with our initial config...
163343-sysmon1331-handles.png


1 Vote 1 ·

Sorry for late respons from my part..
I can understand the problem to recreate this...
If I test a config based only on Hartogs examples everything works just fine...
But as soon as I collect all evt7 and add some filters (about 50 lines) for evt 7 the problem start...

I can absolutely test an experimental symon build if you can supply me with one.. cannot really find an hotmail address to you !?

0 Votes 0 ·

I'm having similar troubles with event 7 and some sort of memory leak - handles don't seem to be increasing though. Just trying to narrow it down right now.

In my case this behavior also exists in Sysmon v10.42 (deployed in production) so i thought i'd try the newer release, which if anything shows the problem much faster.

I'll drop you an email.

0 Votes 0 ·

Finaly had some time over for more testing in this case.....

As mentioned, Hartogs configuration for Evt 7 works for collecting data
But!
when adding more exclude rules (+30 lines) for Evt 7 to the same configuration I see the increased handles count once again.....

Version 13.20 works without any problem with any of our configurations, but from 13.21 - 13.31 the problem exists.
Will try to send in this to sysite(a)microsoft.com a third time.....

0 Votes 0 ·

Did some more testing now with version 13.33..
The problem is still present even in this version...
we are now running a stripped down version of our Evt.7 filter that seems to work on our testsystems..

But!, if we add just a couple of "ImageLoaded" lines to our working config, pointing out our EndpointAV software, the problem is back again..

Ex.
<ImageLoaded condition="begin with">C:\Program Files (x86)\<Endpoint-AV-version>\</ImageLoaded>

The more lines we add like the one above, the faster the climb of handles...

0 Votes 0 ·
Show more comments