Built-in IPSec/IKE2 connection issue

Question

Hi,

struggling with proper configuration of IPSec/IKE2 VPN tunnels on Win10 to MikroTik RB4011 routers.

Windows 10 Pro [Version 10.0.19041.508]

network configuration:
Internet ----- <public ip> media gateway <192.168.0.1/24> ----- <192.168.0.7/24> RB4011 <internal 192.xxxxxx subnets, 10.1.2.0/24 as VPN subnet>

CA and the server and client certs are all issued by the MikrotTik router and self-signed.
Using machine certificate in Win, CA and client cert are installed in the machine cert store.

Accesses with iOS and Android are working to all routers, have a stable connection. Can access the routers and subnets behind them over the VPN channels.

I've tried the NCP Secure Entry Client, which connects from the very same Win10 client where the Windows' build-in does not.

Windows 10 client is not working, but is showing (same) strange behavior towards all routers.
All RB4011 say that a connection with the Win 10 machine is established (SA established, peer authorized). After the DPD timeout they kill the given SA.
Windows 10 throws an error message: IKE2 credentials are unacceptable. The Event viewer shows an 13801 error code.

That leads me think that the problem lies within the Windows solution.

What should be done, so that the issue is resolved and such IPSec/IKE2 connections can be established with the built-in VPN client?

if anyone would like to read on for further details in posts / few comments in the MikroTik forum:
https://forum.mikrotik.com/viewtopic.php?t=165673
https://forum.mikrotik.com/viewtopic.php?t=164982

log from the Mikrotik:
17:31:04 ipsec ipsec::: -> ike2 request, exchange: SA_INIT:0 <client public IP>[54774] 56cc6ef715838de1:0000000000000000
17:31:04 ipsec ipsec::: ike2 respond
17:31:04 ipsec ipsec::: payload seen: SA
17:31:04 ipsec ipsec::: payload seen: KE
17:31:04 ipsec ipsec::: payload seen: NONCE
17:31:04 ipsec ipsec::: payload seen: NOTIFY
17:31:04 ipsec ipsec::: payload seen: NOTIFY
17:31:04 ipsec ipsec::: payload seen: NOTIFY
17:31:04 ipsec ipsec::: payload seen: VID
17:31:04 ipsec ipsec::: payload seen: VID
17:31:04 ipsec ipsec::: payload seen: VID
17:31:04 ipsec ipsec::: payload seen: VID
17:31:04 ipsec ipsec::: processing payload: NONCE
17:31:04 ipsec ipsec::: processing payload: SA
17:31:04 ipsec ipsec::: IKE Protocol: IKE
17:31:04 ipsec ipsec::: proposal #1
17:31:04 ipsec ipsec::: enc: 3des-cbc
17:31:04 ipsec ipsec::: prf: hmac-sha1
17:31:04 ipsec ipsec::: auth: sha1
17:31:04 ipsec ipsec::: dh: modp1024
17:31:04 ipsec ipsec::: proposal #2
17:31:04 ipsec ipsec::: enc: 3des-cbc
17:31:04 ipsec ipsec::: prf: hmac-sha256
17:31:04 ipsec ipsec::: auth: sha256
17:31:04 ipsec ipsec::: dh: modp1024
17:31:04 ipsec ipsec::: proposal #3
17:31:04 ipsec ipsec::: enc: 3des-cbc
17:31:04 ipsec ipsec::: prf: unknown
17:31:04 ipsec ipsec::: auth: unknown
17:31:04 ipsec ipsec::: dh: modp1024
17:31:04 ipsec ipsec::: proposal #4
17:31:04 ipsec ipsec::: enc: aes128-cbc
17:31:04 ipsec ipsec::: prf: hmac-sha1
17:31:04 ipsec ipsec::: auth: sha1
17:31:04 ipsec ipsec::: dh: modp1024
17:31:04 ipsec ipsec::: proposal #5
17:31:04 ipsec ipsec::: enc: aes128-cbc
17:31:04 ipsec ipsec::: prf: hmac-sha256
17:31:04 ipsec ipsec::: auth: sha256
17:31:04 ipsec ipsec::: dh: modp1024
17:31:04 ipsec ipsec::: proposal #6
17:31:04 ipsec ipsec::: enc: aes128-cbc
17:31:04 ipsec ipsec::: prf: unknown
17:31:04 ipsec ipsec::: auth: unknown
17:31:04 ipsec ipsec::: dh: modp1024
17:31:04 ipsec ipsec::: proposal #7
17:31:04 ipsec ipsec::: enc: aes192-cbc
17:31:04 ipsec ipsec::: prf: hmac-sha1
17:31:04 ipsec ipsec::: auth: sha1
17:31:04 ipsec ipsec::: dh: modp1024
17:31:04 ipsec ipsec::: proposal #8
17:31:04 ipsec ipsec::: enc: aes192-cbc
17:31:04 ipsec ipsec::: prf: hmac-sha256
17:31:04 ipsec ipsec::: auth: sha256
17:31:04 ipsec ipsec::: dh: modp1024
17:31:04 ipsec ipsec::: proposal #9
17:31:04 ipsec ipsec::: enc: aes192-cbc
17:31:04 ipsec ipsec::: prf: unknown
17:31:04 ipsec ipsec::: auth: unknown
17:31:04 ipsec ipsec::: dh: modp1024
17:31:04 ipsec ipsec::: proposal #10
17:31:04 ipsec ipsec::: enc: aes256-cbc
17:31:04 ipsec ipsec::: prf: hmac-sha1
17:31:04 ipsec ipsec::: auth: sha1
17:31:04 ipsec ipsec::: dh: modp1024
17:31:04 ipsec ipsec::: proposal #11
17:31:04 ipsec ipsec::: enc: aes256-cbc
17:31:04 ipsec ipsec::: prf: hmac-sha256
17:31:04 ipsec ipsec::: auth: sha256
17:31:04 ipsec ipsec::: dh: modp1024
17:31:04 ipsec ipsec::: proposal #12
17:31:04 ipsec ipsec::: enc: aes256-cbc
17:31:04 ipsec ipsec::: prf: unknown
17:31:04 ipsec ipsec::: auth: unknown
17:31:04 ipsec ipsec::: dh: modp1024
17:31:04 ipsec ipsec::: proposal #13
17:31:04 ipsec ipsec::: enc: aes128-gcm
17:31:04 ipsec ipsec::: prf: hmac-sha1
17:31:04 ipsec ipsec::: dh: modp1024
17:31:04 ipsec ipsec::: proposal #14
17:31:04 ipsec ipsec::: enc: aes128-gcm
17:31:04 ipsec ipsec::: prf: hmac-sha256
17:31:04 ipsec ipsec::: dh: modp1024
17:31:04 ipsec ipsec::: proposal #15
17:31:04 ipsec ipsec::: enc: aes128-gcm
17:31:04 ipsec ipsec::: prf: unknown
17:31:04 ipsec ipsec::: dh: modp1024
17:31:04 ipsec ipsec::: proposal #16
17:31:04 ipsec ipsec::: enc: aes256-gcm
17:31:04 ipsec ipsec::: prf: hmac-sha1
17:31:04 ipsec ipsec::: dh: modp1024
17:31:04 ipsec ipsec::: proposal #17
17:31:04 ipsec ipsec::: enc: aes256-gcm
17:31:04 ipsec ipsec::: prf: hmac-sha256
17:31:04 ipsec ipsec::: dh: modp1024
17:31:04 ipsec ipsec::: proposal #18
17:31:04 ipsec ipsec::: enc: aes256-gcm
17:31:04 ipsec ipsec::: prf: unknown
17:31:04 ipsec ipsec::: dh: modp1024
17:31:04 ipsec ipsec::: matched proposal:
17:31:04 ipsec ipsec::: proposal #11
17:31:04 ipsec ipsec::: enc: aes256-cbc
17:31:04 ipsec ipsec::: prf: hmac-sha256
17:31:04 ipsec ipsec::: auth: sha256
17:31:04 ipsec ipsec::: dh: modp1024
17:31:04 ipsec ipsec::: processing payload: KE
17:31:04 ipsec ipsec::: adding payload: SA
17:31:04 ipsec ipsec::: adding payload: KE
17:31:04 ipsec ipsec::: adding payload: NONCE
17:31:04 ipsec ipsec::: adding notify: NAT_DETECTION_SOURCE_IP
17:31:04 ipsec ipsec::: adding notify: NAT_DETECTION_DESTINATION_IP
17:31:04 ipsec ipsec::: adding payload: CERTREQ
17:31:04 ipsec ipsec::: <- ike2 reply, exchange: SA_INIT:0 <client public IP>[54774] 56cc6ef715838de1:04115b535d674c3e
17:31:04 ipsec,info new ike2 SA (R): 192.168.0.7[500]-<client public IP>[54774] spi:04115b535d674c3e:56cc6ef715838de1
17:31:04 ipsec,info ipsec::: new ike2 SA (R): 192.168.0.7[500]-<client public IP>[54774] spi:04115b535d674c3e:56cc6ef715838de1
17:31:04 ipsec ipsec::: processing payloads: VID
17:31:04 ipsec ipsec::: peer is MS Windows (ISAKMPOAKLEY 9)
17:31:04 ipsec ipsec::: processing payloads: NOTIFY
17:31:04 ipsec ipsec::: notify: IKEV2_FRAGMENTATION_SUPPORTED
17:31:04 ipsec ipsec::: notify: NAT_DETECTION_SOURCE_IP
17:31:04 ipsec ipsec::: notify: NAT_DETECTION_DESTINATION_IP
17:31:04 ipsec ipsec::: (NAT-T) REMOTE LOCAL
17:31:04 ipsec ipsec::: KA list add: 192.168.0.7[4500]-><client public IP>[54774]
17:31:06 ipsec ipsec::: -> ike2 request, exchange: AUTH:1 <client public IP>[54783] 56cc6ef715838de1:04115b535d674c3e
17:31:06 ipsec ipsec::: peer ports changed: 54774 -> 54783
17:31:06 ipsec ipsec::: KA remove: 192.168.0.7[4500]-><client public IP>[54774]
17:31:06 ipsec ipsec::: KA list add: 192.168.0.7[4500]-><client public IP>[54783]
17:31:06 ipsec ipsec::: payload seen: ENC
17:31:06 ipsec ipsec::: processing payload: ENC
17:31:06 ipsec ipsec::: payload seen: ID_I
17:31:06 ipsec ipsec::: payload seen: CERT
17:31:06 ipsec ipsec::: payload seen: CERTREQ
17:31:06 ipsec ipsec::: payload seen: AUTH
17:31:06 ipsec ipsec::: payload seen: CONFIG
17:31:06 ipsec ipsec::: payload seen: SA
17:31:06 ipsec ipsec::: payload seen: TS_I
17:31:06 ipsec ipsec::: payload seen: TS_R
17:31:06 ipsec ipsec::: processing payloads: NOTIFY (none found)
17:31:06 ipsec ipsec::: ike auth: respond
17:31:06 ipsec ipsec::: processing payload: ID_I
17:31:06 ipsec ipsec::: ID_I (DER DN): CN=client-Dell7250-2.city,C=HU,ST=County,L=City,O=Home,OU=home,SN=
17:31:06 ipsec ipsec::: processing payload: ID_R (not found)
17:31:06 ipsec ipsec::: processing payload: AUTH
17:31:06 ipsec ipsec::: processing payload: CERT
17:31:06 ipsec ipsec::: got CERT: CN=client-Dell7250-2.city,C=HU,ST=County,L=City,O=Home,OU=home,SN=
17:31:06 ipsec ipsec::: processing payloads: NOTIFY (none found)
17:31:06 ipsec ipsec::: processing payload: AUTH
17:31:06 ipsec ipsec::: requested auth method: RSA
17:31:06 ipsec,info,account peer authorized: 192.168.0.7[4500]-<client public IP>[54783] spi:04115b535d674c3e:56cc6ef715838de1
17:31:06 ipsec,info,account ipsec::: peer authorized: 192.168.0.7[4500]-<client public IP>[54783] spi:04115b535d674c3e:56cc6ef715838de1
17:31:06 ipsec ipsec::: processing payloads: NOTIFY (none found)
17:31:06 ipsec ipsec::: peer wants tunnel mode
17:31:06 ipsec ipsec::: processing payload: CONFIG
17:31:06 ipsec ipsec::: attribute: internal IPv4 address
17:31:06 ipsec ipsec::: attribute: internal IPv4 DNS
17:31:06 ipsec ipsec::: attribute: internal IPv4 NBNS
17:31:06 ipsec ipsec::: attribute: MS internal IPv4 server
17:31:06 ipsec,info acquired 10.1.2.105 address for <client public IP>, CN=client-Dell7250-2.city,C=HU,ST=County,L=City,O=Home,OU=home,SN=
17:31:06 ipsec,info ipsec::: acquired 10.1.2.105 address for <client public IP>, CN=client-Dell7250-2.city,C=HU,ST=County,L=City,O=Home,OU=home,SN=
17:31:06 ipsec ipsec::: processing payload: TS_I
17:31:06 ipsec ipsec::: 0.0.0.0/0
17:31:06 ipsec ipsec::: [::/0]
17:31:06 ipsec ipsec::: processing payload: TS_R
17:31:06 ipsec ipsec::: 0.0.0.0/0
17:31:06 ipsec ipsec::: [::/0]
17:31:06 ipsec ipsec::: TSi in tunnel mode replaced with config address: 10.1.2.105
17:31:06 ipsec ipsec::: canditate selectors: 0.0.0.0/0 <=> 10.1.2.105
17:31:06 ipsec ipsec::: canditate selectors: [::/0] <=> [::/0]
17:31:06 ipsec ipsec::: processing payload: SA
17:31:06 ipsec ipsec::: IKE Protocol: ESP
17:31:06 ipsec ipsec::: proposal #1
17:31:06 ipsec ipsec::: enc: aes256-cbc
17:31:06 ipsec ipsec::: auth: sha1
17:31:06 ipsec ipsec::: proposal #2
17:31:06 ipsec ipsec::: enc: aes128-cbc
17:31:06 ipsec ipsec::: auth: sha1
17:31:06 ipsec ipsec::: proposal #3
17:31:06 ipsec ipsec::: enc: 3des-cbc
17:31:06 ipsec ipsec::: auth: sha1
17:31:06 ipsec ipsec::: proposal #4
17:31:06 ipsec ipsec::: enc: des-cbc
17:31:06 ipsec ipsec::: auth: sha1
17:31:06 ipsec ipsec::: proposal #5
17:31:06 ipsec ipsec::: enc: null
17:31:06 ipsec ipsec::: auth: sha1
17:31:06 ipsec ipsec::: searching for policy for selector: 0.0.0.0/0 <=> 10.1.2.105
17:31:06 ipsec ipsec::: generating policy
17:31:06 ipsec ipsec::: matched proposal:
17:31:06 ipsec ipsec::: proposal #1
17:31:06 ipsec ipsec::: enc: aes256-cbc
17:31:06 ipsec ipsec::: auth: sha1
17:31:06 ipsec ipsec::: ike auth: finish
17:31:06 ipsec ipsec::: ID_R (FQDN): RB4011-2020.city
17:31:06 ipsec ipsec::: processing payload: NONCE
17:31:06 ipsec ipsec::: cert: CN=RB4011-2020.city,C=HU,ST=County,L=City,O=Home,OU=home,SN=
17:31:06 ipsec ipsec::: adding payload: CERT
17:31:06 ipsec ipsec::: adding payload: ID_R
17:31:06 ipsec ipsec::: adding payload: AUTH
17:31:06 ipsec ipsec::: adding notify: INITIAL_CONTACT
17:31:06 ipsec ipsec::: preparing internal IPv4 address
17:31:06 ipsec ipsec::: preparing internal IPv4 netmask
17:31:06 ipsec ipsec::: preparing internal IPv6 subnet
17:31:06 ipsec ipsec::: preparing internal IPv4 DNS
17:31:06 ipsec ipsec::: preparing internal IPv4 DNS
17:31:06 ipsec ipsec::: preparing internal IPv4 DNS
17:31:06 ipsec ipsec::: preparing internal IPv4 DNS
17:31:06 ipsec ipsec::: adding payload: CONFIG
17:31:06 ipsec ipsec::: initiator selector: 10.1.2.105
17:31:06 ipsec ipsec::: adding payload: TS_I
17:31:06 ipsec ipsec::: responder selector: 0.0.0.0/0
17:31:06 ipsec ipsec::: adding payload: TS_R
17:31:06 ipsec ipsec::: adding payload: SA
17:31:06 ipsec ipsec::: <- ike2 reply, exchange: AUTH:1 <client public IP>[54783] 56cc6ef715838de1:04115b535d674c3e
17:31:06 ipsec ipsec::: IPsec-SA established: <client public IP>[54783]->192.168.0.7[4500] spi=0x47a46b1
17:31:06 ipsec ipsec::: IPsec-SA established: 192.168.0.7[4500]-><client public IP>[54783] spi=0x58562590
<<------->>
17:33:06 ipsec ipsec::: sending dpd packet
17:33:06 ipsec ipsec::: <- ike2 request, exchange: INFORMATIONAL:0 <client public IP>[54783] 56cc6ef715838de1:04115b535d674c3e
17:33:11 ipsec ipsec::: dpd: retransmit
17:33:16 ipsec ipsec::: dpd: retransmit
17:33:21 ipsec ipsec::: dpd: retransmit
17:33:26 ipsec ipsec::: dpd: retransmit
17:33:31 ipsec ipsec::: dpd: max retransmit failures reached
17:33:31 ipsec,info killing ike2 SA: 192.168.0.7[4500]-<client public IP>[54783] spi:04115b535d674c3e:56cc6ef715838de1
17:33:31 ipsec,info ipsec::: killing ike2 SA: 192.168.0.7[4500]-<client public IP>[54783] spi:04115b535d674c3e:56cc6ef715838de1
17:33:31 ipsec ipsec::: IPsec-SA killing: <client public IP>[54783]->192.168.0.7[4500] spi=0x47a46b1
17:33:31 ipsec ipsec::: IPsec-SA killing: 192.168.0.7[4500]-><client public IP>[54783] spi=0x58562590
17:33:31 ipsec ipsec::: removing generated policy
17:33:31 ipsec ipsec::: adding payload: DELETE
17:33:31 ipsec ipsec::: <- ike2 request, exchange: INFORMATIONAL:1 <client public IP>[54783] 56cc6ef715838de1:04115b535d674c3e
17:33:31 ipsec ipsec::: KA remove: 192.168.0.7[4500]-><client public IP>[54783]
17:33:31 ipsec,info releasing address 10.1.2.105
17:33:31 ipsec,info ipsec::: releasing address 10.1.2.105

Answer 1

Hi,

issue solved, though through other sources.

I did not dig into the very deep details of the white papers on the IPSec solution, though the Microsoft implementation I see more restrictive than others, as any other combo (Android, iOS, 3rd party client on Win10) was working with this type of router, with same setup and certs generated the same way / same usage.

1/ The machine certificate, which is used for IKEv2 validation on the RAS Server, does not have Server Authentication as the EKU (Enhanced Key Usage).
--> It had 'digital signature'
After I recreated all certs (CA, server, client), it now has 'tls server'
It did not and does not have 'IP security IKE intermediate' at all.
This still doesn't match, though the connection can be now established.

2/ The machine certificate on RAS server has expired.
--> it has not

3/ The root certificate to validate the RAS server certificate is not present on the client.
--> it was present

4/ The VPN Server Name, provided on the client, does not match with the subjectName of the server certificate.
--> it matched

5/ Make sure you have selected 'Server Authentication' and 'IP Security IKE intermediate' in Extended Key Usage (EKU).
--> see above (1st)
--> Guess this is the cert on your RAS server. I don't know how the Microsoft solution is checking the server's cert EKU on the client side, if that is sent with the cert itself to the peer from the VPN gateway.

6/ Make sure you have selected the Digital signature and Key encipherment in Key Usage.
--> it was 'digital signature' , now it's 'tls client'. It works w/o 'key encipherment', no need for that. so the previous client cert must had been OK as well.

7/ Make sure the machine certificate on the RAS server has not expired.
--> it did not

8/ Make sure common name is the same as the hostname which is configured as the VPN destination on the VPN client.
--> it was set as RB4011-2020.city which I placed to Win's host file and thus the hostname resolved to the correct address, same as CN. In the attached logs it can be seen that it matched
17:31:06 ipsec ipsec::: ID_R (FQDN): RB4011-2020.city
17:31:06 ipsec ipsec::: processing payload: NONCE
17:31:06 ipsec ipsec::: cert: CN=RB4011-2020.city,C=HU,ST=County,L=City,O=Home,OU=home,SN=

Answer 2

Hi ,

Please understand, we have no such third-party device to test in our lab. I used RRAS and Microsoft CA with windows build-in VPN client, IPSEC VPN connection works fine. As you can see below:

About the issue of the 13801: IKE authentication credentials are unacceptable error, common causes for this issue are:

• The machine certificate, which is used for IKEv2 validation on the RAS Server, does not have Server Authentication as the EKU (Enhanced Key Usage).
• The machine certificate on RAS server has expired.
• The root certificate to validate the RAS server certificate is not present on the client.
• The VPN Server Name, provided on the client, does not match with the subjectName of the server certificate.

For your reference:

Error 13801, IKE authentication credentials are unacceptable

Please Note: Since the web site is not hosted by Microsoft, the link may change without notice. Microsoft does not guarantee the accuracy of this information.

Ensure that the following requirements are met, before you try to establish the IKEV2 connection:

• Make sure you have selected 'Server Authentication' and 'IP Security IKE intermediate' in Extended Key Usage (EKU).

• Make sure you have selected the Digital signature and Key encipherment in Key Usage.

• Make sure the machine certificate on the RAS server has not expired.
• Make sure common name is the same as the hostname which is configured as the VPN destination on the VPN client.

Best Regards,
Candy

--------------------------------------------------------------

If the Answer is helpful, please click "Accept Answer" and upvote it.