The L2TP connection attempt failed because the security layer encountered a processing error during initial negotiations with the remote computer Problem

Question

Hi All, I have recently deploying a windows server 2019 standard with RRAS enabled, when i tired to connect via L2TP vpn dial up icon on windows 10 machine internally, i received the following warning, i have google and people mention the update KB5009543 casued such issue, but i have checked i dont have such updated installed on the windows 10 computer, for the VPN setting, i have tried to select PAP, CHAP and CHAPv2 also no luck, any help would be appreicated

warning:

The L2TP connection attempt failed because the security layer encountered a processing error during initial negotiations with the remote computer

RRAS installation video:

https://www.youtube.com/watch?v=xOyS1gZ0Lrg&t=215s

Answer 1

i figured out is an issue with the following in vmware, thanks for your advise

https://communities.vmware.com/t5/ESXi-Discussions/esxi-5-5-and-microsoft-vpn/m-p/1307615

Answer 2

Hellokeith li1

There are few basic troubleshooting steps you might consider regarding this issue:

1. Check Server Ports and Configuration:

   - Confirm that necessary ports (UDP 500, UDP 4500, and TCP 1701) are open on the server's firewall.

2. Verify L2TP Security Settings:

   - Match security settings on both the Windows Server 2019 RRAS and Windows 10 client:

     - Encryption type (e.g., "Require encryption" or specific protocols like AES).

     - Authentication method (e.g., PAP, CHAP, or CHAPv2).

     - L2TP/IPsec or L2TP with pre-shared key (PSK) settings.

3. Test with Another Client Machine:

   - Attempt a connection from a different Windows 10 machine within the same network to identify if the issue is client-specific or server configuration-related.

4. Check Certificates (if used):

   - If using certificates for L2TP/IPsec authentication, verify their validity and correct configuration on both server and client.

Best Regards,

Karlie Weng