FileSystemWatcher running in webjob and monitoring mounted storage yields random buffer overflow

Question

Hello,
I am experiencing strange behaviors with a webjob that is monitoring a mounted azure file share. The application uses System.IO.FileSystemWatcher to monitor a folder in the mounted file share and process new files by copying them to a different directory.

I am seeing FileSystemWatcher InternalBufferOverflowExceptions either when copying a few files to the file share or randomly at night

There are several observations I have made

1. When copying files to the share, the InternalBufferOverflowExceptions occurs around 15 files no matter on the size of InternalBufferSize. Locally this works with a large number of files, tested over a file share within the local network as well. Why doesn't the increase of the InternalBufferSize allow for more files? I've went up to 65536 (https://learn.microsoft.com/en-us/dotnet/api/system.io.filesystemwatcher.internalbuffersize?view=net-8.0) and the behavior is identical to the default value of 8192
2. Sometimes I see the exception at strange hours during the night such as 5AM of 6AM when I know for certain there isn't anyone using the system. I implemented a very basic C++ app that monitors the same directory in parallel using ReadDirectoryChangesW (which is what FileSystemWatcher uses underneath the hood) and indeed I see a lot of notifications coming at that time during the night for files that are no longer there but were copied during the day . Any ideea why this happens?
3. Once I got the exception a few minutes after copying just one file.

We've tried moving to a premium File Share to use SMB multichannel but it didn't make any difference.

For pt.1 we can live with a limitation, but the most concerning is point nr 2 which I do not understand.

Answer 1

Hello DUMITRASCU Claudiu-Alexandru,  

Thank you for the question. You have posted the query on Azure Storage forums. For specialized assistance on a .NET query, kindly just repost the question on Developer Community to receive focused and immediate assistance from the right set of experts. However, based on my understanding of the issue, I am sharing a few suggestions.

The documentation for FileSystemWatcher states:

The system notifies your component of file changes in a buffer created by the FileSystemWatcher. If there are many changes in a short time, the buffer can overflow. This causes the component to lose track of changes in the directory, and it will only provide blanket notification. Increasing the size of the buffer with the InternalBufferSize property is expensive, as it comes from non-paged memory that cannot be swapped out to disk, so keep the buffer as small yet large enough to not miss any file change events. To avoid a buffer overflow, use NotifyFilter and IncludeSubdirectories properties so you can filter out unwanted change notifications.  

Here is a discussion on similar issue that might help: https://stackoverflow.com/questions/75832346/filesystemwatcher-throws-system-io-internalbufferoverflowexception-in-c-sharp-wi

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.