IPSEC-ESP authentication problem on firewall policy

Question

Hello
I have a Windows 10 client, in Windows Server 2016/2019 domain (Domain A) that needs to connect in RDP (with IPSEC Policy) on a Windows Server 2008 R2 DC in other domain (Domain B).
A one-way trust relationship exists between the domains.
A firewall policy was created on DC 2008R2 that forces kerberos (user + computer) authentication (IPSEC ESP) to allow access to only authorized computers and users.
Both authorized users and clients expressed as domain groups are included in the policy.
Both the client and the DC target have the same Security Rules enabled.
Server and client are on the same network.

When I connect from the windows 10 client I get an error: "An IPsec main mode negotiation failed. ERROR ID 4653"

Local Endpoint:
Local Principal Name: -
Network Address: 10.0.0.99
Keying Module Port: 500

Remote Endpoint:
Principal Name: -
Network Address: 10.0.0.12
Keying Module Port: 500

Additional Information:
Keying Module Name: AuthIP
Authentication Method: Unknown authentication
Role: Responder
Impersonation State: Not enabled
Main Mode Filter ID: 76654

Failure Information:
Failure Point: Remote computer
Failure Reason: IKE authentication credentials are unacceptable
State: Sent second (KE) payload
Initiator Cookie: 560a805cd9eb9131
Responder Cookie: 1d0925bef6305e21

I cannot find any information or solution for this error. Can you give me some tips?

Answer 1

Hi there,

Events generated by the Authenticated Internet Protocol (AuthIP) and the Internet Key Exchange protocol (IKE) during Main Mode negotiations are audited by the codes which fall under the Audit IPsec Main Mode subcategory. Event 4653 falls under this category.

Here is a troubleshooting link for your reference.IPSec Troubleshooting http://technet.microsoft.com/en-us/library/cc783041(v=WS.10).aspx

Troubleshooting VPN over IPsec http://technet.microsoft.com/en-us/library/bb794765.aspx

The below thread discusses the same issue and you can try out some troubleshooting steps from this and see if that helps you to sort the Issue.

Audit Failure - Event ID 4653 - An IPsec main mode negotiation failed https://social.technet.microsoft.com/Forums/en-US/5c5befdb-db83-42e9-8ede-85d049806303/audit-failure-event-id-4653-an-ipsec-main-mode-negotiation-failed?forum=winserversecurity

Audit IPsec Main Mode https://learn.microsoft.com/en-us/windows/security/threat-protection/auditing/audit-ipsec-main-mode

---------------------------------------------------------------------------------------------------------------------------------------

--If the reply is helpful, please Upvote and Accept it as an answer–