Hello Frederic,
One temporary protection measure might be to reduce the limit on the number of nested windows. By default, the limit is 50 but it can be reduced by editing the registry value USERNestedWindowLimit
under the key HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Windows
. You might need to use trial and error to determine the maximum value that is safe on your (and your customers) system (it might vary depending on what else requires space on the kernel stack (e.g. some security products)). I would suggest a value of about 30 (rough calculations suggest that about 31 would be the limit, but that would be sailing close to the wind).
[Update: only values between 35 and 100 can be set as the value for USERNestedWindowLimit, but I think that 30 to 31 would be needed to prevent the issue]
Obviously, applications that try to nest windows deeply might fail (or fail to work properly).
It might be possible to increase the kernel stack size instead of reducing the window nesting depth - I am still investigating this. [Update: there appears to be no practical method of influencing this value; it is calculated at boot time based on characteristics of the CPU].
Nesting windows deeper than 35 is possible, but one would have to ensure that some layers of nested windows are explicitly deleted before allowing the top-level window to be deleted - it is the recursive deletion that causes the kernel stack to overflow.
UPDATE: see https://support.microsoft.com/en-au/topic/february-13-2024-kb5034768-os-build-17763-5458-f06ecec0-23b0-4860-880b-74a2fd1b56c0:
This update addresses an issue that affects the Windows nesting limit. You can now set it to a low of 35 instead of 50. This is the registry value in HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Windows\USERNestedWindowLimit. To learn more, see What is the window nesting limit?. Do not change this limit unless a kernel stack overflow and a recursion in DestroyWindow() cause stop errors.
Gary