RRAS server for AOVPN consuming all memory

Question

I have several Routing and Remote Access (RRAS) servers setup for AOVPN, and have been experiencing a problem where they're consuming all of the system's memory. I'm hoping to see if anyone else has experienced this. I'm seeing this on the instance of svchost.exe that's running from the command line "C:\Windows\system32\svchost.exe -k netsvcs -p -s IKEEXT", so the IKE and AuthIP IPsec Keying Modules service. After the server is started, the service starts adding about 20-40KB of memory usage per second until it has consumed it all.

Additional relevant information:

• The affected servers are all running Server 2022 Core.
• List item
• This problem never happens on Server 2019 Core. Servers running a completely identical configuration with 2019 are OK, but the 2022 servers are all affected.
• List item
• I've done several tests swapping out 2019 and 2022 for the same hosts with the same result of 2019 always being fine, and 2022 always being broken.
• List item
• I've tried playing with basically any setting I can think of, for example, changing my custom IPSec properties with Set-VpnServerConfiguration, and everything I can think of had no effect.
• List item
• All ports on RRAS are disabled with the exception of the IKEV2 ports.
• List item
• Once the memory usage has gotten sufficiently high, stopping or restarting the RemoteAccess service becomes impossible, and rebooting the server is the only way to clear it.
• List item
• I see this only on my production systems that have a large number of clients, but not on testing servers that have very few clients.
• List item
• The servers all have 4 cores and 16 GB RAM, all of which is all eventually consumed by the service.

I'm considering returning everything to Server 2019, but I want to see if there's a fix for Server 2022 before doing so.

Answer 1

This is now resolved. The problem went away after applying the September 2022 update (KB5017316). When we looked at the list of files changed in that update, IKEEXT.DLL was among them, so I assume that's what fixed it.