question

DenisPayne-4809 avatar image
4 Votes"
DenisPayne-4809 asked Bartek82 commented

Windows Defender creating thousands of files

Since 28/04/2021 around 22:00, thousands of files started to be created in folder C:\ProgramData\Microsoft\Windows Defender\Scans\History\Store\ on one of my domain controllers.

There were over 200k files which caused that night's backup to take over 4hours rather then the normal 20minutes.
There are now well over 400k files.

Another member server is also affected by this, there are over 2million files in the same folder being:
C:\ProgramData\Microsoft\Windows Defender\Scans\History\Store\

Bother servers are running Windows Server 2016.
The files are 1-2KB.
Settings>Update&Security>Windows Defender settings are enabled.
Windows Defender GUI>History is empty for Quarantined, Allowed and All Detected items.
No Windows Defender scan is running.

Resource Monitor>Disk>Disk Activity shows the System process accessing these files, so I presume it is creating them.
System is also the owner of these files.

windows-serverwindows-server-2016windows-server-security
· 12
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Same here since Friday!
We have Sophos and datto RMM on the systems.
Any news now?

0 Votes 0 ·

Have the same problem. Over the weekend, 3 server(2019) hard drives filled up.

We also use Sophos Central Endpoint

0 Votes 0 ·

We are also facing this issue as well. We are on Windows Defender Engine version 1.1.18100.5 with Sophos installed. My colleague came across this Reddit thread (https://www.reddit.com/r/sysadmin/comments/n0q8pc/help_windows_defender_real_time_protection/) dealing with the same issue. Looks like an update to the engine to bring it to version 1.1.18100.6 may have resolved it for a few of those people but I don't think it's publically available yet. I've had at least 10 servers already affected by this and all of the issues started on 4/28. Hooray for Mondays!

0 Votes 0 ·

Has anyone got any resolution from Microsoft on this issue yet? If so, is there an update that's publically available?

0 Votes 0 ·

I got a response that a tech has been assigned to my case, that's it. You can temporarily disable realtime protection to stop file creation, or remove WD in Apps & Features.

0 Votes 0 ·

In WS2016 it wasn't possible to disable Windows Defender, if you did then flipped away then back to the screen it'd just be running again.
In my case something about 'Unable to disable Windows Defender if Sophos installed' so I just uninstalled Windows Defender from the 3xWS 2016 VMs that had the issue.

Supposed an update later this week will fix Windows Defender issue which is the cause.

0 Votes 0 ·
Show more comments
DenisPayne-4809 avatar image
0 Votes"
DenisPayne-4809 answered

All in 7xWS2016 servers all running Sophos were affected, across two of my clients.
Windows Defender was thus uninstalled from 3xservers with small C-Drives to prevent 0% free space issue.

Cause seems to of been a MSFT Windows Defender update for which a fix was released late last week.

Windows Defender has been re-installed on the 3xservers it was previously uninstalled from.
None of the 7xWS2016 servers are showing a repeat of the issue so assume MSFT fixed it with a Windows Defender update.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

DSPatrick avatar image
0 Votes"
DSPatrick answered DenisPayne-4809 commented

Something here may help.
https://support.microsoft.com/en-us/windows/protection-history-f1e5fd95-09b4-46d1-b8c7-1059a1e09708

--please don't forget to Accept as answer if the reply is helpful--



· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

WS 2016 so protection history is in the seperate Windows Defender GUI, not yet integrated.

There are no detections, there is no scan running, just hundreds of thousands of 1-2KB files being created.

0 Votes 0 ·
PaulMartin-1885 avatar image
0 Votes"
PaulMartin-1885 answered LeeW-6772 commented

Just to add that we have also seen this issue which looks to be across multiple servers.
An example server has the folder is almost 2 million files large with the majority created from 29th April

Server 2016
Windows Defender Versions

Antimalware Client: 4.18.2001.7
Engine Version: 1.1.18100.5
Antivirus Definitions: 1.337.307.0
Network inspection system engine version: 1.1.18100.5
Network inspection system definition versions: 1.337.307.0

EDIT: We're also running Sophos on the impacted machines, I've raised a ticket with Sophos to see if they can check their side too or re-create the issue
After checking some servers though it seemed to start after a definitions update for Windows Defender after the MpKslacab service was re-installed
Between impacted servers, the "Engine Version" of Windows Defender seems to be the only one that matches other impacted servers too

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

I've got this issue on 2 servers with the following

Antimalware client version: 4.18.2103.7
Engine version: 1.1.18100.5
Antivirus definition: 1.337.615.0
Network inspection system engine version: 1.1.18100.5
Network inspection system definition version: 1.337.615.0

I've checked other servers in our estate and the Engine version is 1.1.18100.6, these don't seem to be affected.

As the folder structure for Defender is locked updating it via Windows Updates isn't working, I'm assuming the files can't be accessed. I'm going to try templating one of our broken servers, removing the Defender feature, rebooting and then adding the feature again to see if that fixes the issue.

0 Votes 0 ·
DavidFosbenner-1768 avatar image
0 Votes"
DavidFosbenner-1768 answered DenisPayne-4809 commented

Oh thank goodness someone else has this issue! I thought I was losing my mind.

Starting on 4/29, 2 of my file servers suddenly had zero disk space. I'm running Windows Server 2019, I have the same issue with the same Store folder. This folder had about 1 million files, all under 2K, all dated within the last 24 hours. The only way I could stop creation of the files was disabling Defender's real-time protection. The files took up about 4GB. I deleted them all.

Since the servers are virtual machines I added 10GB to each C: drive. Well, guess what? Tonight the disks were full again, this time with over 11GB and 4 million files!

I just opened a case with MS PPI Support. When a server has no disk space things stop working, so obviously this is urgent. For now I've disable real-time protection and deleted the files again.

This is insane! I haven't made any system changes since the last patch Tuesday. I don't know what MS did but this is definitely on them IMO.

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Aye, sounds to be the recent Windows Defender definition update maybe.
Hoping MSFT picks up on this post and likely others to resolve the issue in next update.

0 Votes 0 ·
AndreasSchweizerdivertogmbh-8979 avatar image
0 Votes"
AndreasSchweizerdivertogmbh-8979 answered DenisPayne-4809 commented

Same here on some 2016 servers.
Any news from MS?
We habe Sophso Endpoint and datto RMM.. do you have some similar?

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

We have Sophos Endpoint and Atera RMM.
We were hit by ransomeware several months back so have opened some expensive Sophos support monitoring ticket.
They advised nothing to do with Sophos.

0 Votes 0 ·
DavidFosbenner-1768 avatar image
0 Votes"
DavidFosbenner-1768 answered DenisPayne-4809 commented

Good morning. Yes, I also have Sophos. I didn't get anything from MS yet, but I also opened a ticket with Sophos, asking about Defender AV and Sophos both running on the same system. It's been my experience in the past that when I install AV, it disables the native Windows AV, but not in this case. Here's what Sophos support said: (my comments in [brackets]).

"To assist you with your query Sophos can run with Windows Defender but it's advisable not to run both for we might encounter a performance issue when they run at the same time.

You can check by running command prompt as administrator, and run the fltmc command to see what drives are available on the server. [If fltmc returns "WdFilter" then Defender is running.]

Kindly try to disable the Windows Defender in Manage Roles and Features, untick the Windows Defender feature. [Reboot] and run the fltmc command again to confirm [WdFilter is removed]."

I did this on one server and confirmed Defender AV is no longer installed. It hopefully will solve the issue for me, but it doesn't explain the cause or truly "fix" what is broken. I might leave Defender on one server and work with MS to find the issue.

· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Sophos was very shady with me about it was ok to run Sophos Central Endpoint alongside Windows Defender.

I thus just uninstalled Windows Defender from the 3xaffected WS2016 servers then after a reboot no new ..\Scans\History\Store\\ files being created.

0 Votes 0 ·
JamesFairless-0939 avatar image
0 Votes"
JamesFairless-0939 answered

We are also having the same issue on a Server 2019 VM box running Sophos. First noticed due to a large change in back up length and time. over 1 million new and modified files created in C:\ProgramData\Microsoft\Windows Defender\Scans\History\Store.

Also noticed that CPU usage was pinned with Sophos and Windows defender both being the culprits, starting to look like updates to definitions has both bits of software fighting each other causing the creation of millions of files.

Come on MS we need a support update please.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

DavidFosbenner-1768 avatar image
0 Votes"
DavidFosbenner-1768 answered

While I'm curious to know what caused this, I'm not holding my breath waiting for MS. They'll probably blame Sophos. I have about a dozen servers, 4 had this issue. I uninstalled Defender from 10 servers and left it on 2 servers for testing/troubleshooting purposes. There's no need for it if the Sophos protection is there, so I don't see myself reinstalling Defender even if this issue gets fixed.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

AndrewBounds-3913 avatar image
0 Votes"
AndrewBounds-3913 answered AndrewBounds-3913 edited

2021-05-03:

Problem is with the AMEngineVersion < 1.1.18100.6. New version is supposed to be deployed by Microsoft Thursday May 6th.



I have this issue on a Windows 2008 R2 server running SCEP and a 2016 server running Defender. Do not have Sophos on either.

 Windows 2008 R2:
 C:\ProgramData\Microsoft\Microsoft Antimalware\Scans\History\Store
 Windows 2016:
 C:\ProgramData\Microsoft\Windows Defender\Scans\History\Store
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

YuhanDeng-MSFT avatar image
0 Votes"
YuhanDeng-MSFT answered

Hi,
Based on your description, I did some research but got nothing. To resolve this issue, I would suggest that you contact Microsoft Customer Support and Services where more in-depth investigation can be done so that you would get a more satisfying explanation and solution to this issue. In addition, if the issue has been proved as system flaw, the consulting fee would be refund. You may find phone number for your region accordingly from the link below.
Global Customer Service phone numbers:
https://support.microsoft.com/en-us/help/13948/global-customer-service-phone-numbers

Thanks for your time.
Best regards,
Danny


If the Answer is helpful, please click "Accept Answer" and upvote it.

Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.


5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.