VPN stops working. No ISAKMP packets sent.

Question

VPN stops working. No ISAKMP packets sent.

Jerry 21

My VPN connection on a Win10 machine suddenly stopped working. However, the VPN on a virtual Win10 on the same machine continued to work. Ran thru standard fixes and finally used Wireshark to track packets. I could see complete session on the virtual machine but nothing on the host machine. Did a logman capture and found that Send ISAKMP packet was immediately followed by Packet Dropped - Error Code: IPsec dropped a clear text packet.

SFC fixed a few bad files. No change.
Reinstalled Win10 keeping programs and data. No change.
Problem affects all users on the host machine. Virtual machine unaffected.

The appears to be a problem with some installed program interfering with forming the ISAKMP packet but I have no clue how to proceed from here short of wiping and reinstalling a clean OS.

Accepted answer

4 additional answers

Your answer

Answer 1

Hello @Jerry ,

Many thanks for that (nicely sanitized) data - it was just what I wanted; it is nice to collaborate with someone so competent. The data is also a big step forward in solving the problem.

When no L2TP VPN is active, there are normally no L2TP objects in the WFP state; they are added by RasMan when trying to connect the VPN. Here is a trace from my system that shows the objects that are added at that time:

If you view the image in its original size, you should see that it adds "IKE Exemption" filters and L2TP specific filters and provider context.

If you run a "strings" program (capable of showing UTF-16 strings) over %SystemRoot%\System32\rasmans.dll and %SystemRoot%\System32\mprddm.dll and search for L2TP, you will probably find:

L2TP Server Inbound and Outbound Filter
L2TP Server Outbound Filter
L2TP Server Inbound Filter
L2TP Server Filter1
L2TP Client Inbound and Outbound Filter
L2TP Client Specific Filter
L2TP No Encryption Quick Mode Policy
L2TP Optional Encryption Quick Mode Policy
L2TP Require Encryption Quick Mode Policy
L2TP Require Max Encryption Quick Mode Policy
L2TP Custom Encryption Quick Mode Policy
L2TP Main Mode Policy

What you probably won't find is the string "L2TP UDP ports" (I can't find that anywhere on my system), but there are 54 filters with that name in your WFP state (referring to ports 500, 4500 and 1701).

Where you probably will find the string is in the "Windows Defender Firewall with Advanced Security" MMC snap-in (in both the "Inbound Rules" and "Outbound Rules" lists).

Some person or component (almost certainly not from Microsoft) has added these rules/filters. Just disabling the rules should enable the built-in VPN to work.

If you can identify the source of the rule, you should be able to prevent the problem from reoccurring.

Gary

Answer 2

Gary Nebbett 6,201

Hello @Jerry ,

If you want, you could try the following. Issue the command netsh trace start report=disabled provider=Microsoft-Windows-RRAS provider=Microsoft-Windows-WFP provider={106B464D-8043-46B1-8CB8-E92A0CD7A560} keywords=0x10 level=4 tracefile=why.etl to start a trace, try to connect to the VPN and then issue the command netsh trace stop to stop the trace.

The trace might contain more information than you are comfortable with sharing, but if you are prepared to share the data, make the why.etl file available here via a OneDrive, Google Drive, etc. link and I will take a look at it.

Gary

Jerry 21 Reputation points

2021-06-09T18:19:21.817+00:00

The Google drive link keeps getting blocked when I try to make the file available
Gary Nebbett 6,201 Reputation points

2021-06-09T18:32:27.75+00:00

Hello @Jerry ,

Your last message seems a bit ambiguous to me. Are you saying that this site (https://learn.microsoft.com/en-us/answers/...) will not let you include a URL in a post or that Google Drive won't generate a shareable URL for you?

If the .etl file is small enough (less than 3.1MB, I think), perhaps after zipping the file to reduce its size (if necessary), you could try renaming the file to one of the accepted extensions (xml, gif, png, jpg, jpeg, pdf, txt, log) and attaching it to a message.

Gary

Answer 3

Gary Nebbett 6,201

Hello @Jerry ,

The link works, but you should delete it now - buried deep within the trace is something that should not be shared.

A trace of L2TP/IPsec using the Windows VPN client starts with lots of events from AgileVPN/RRAS (the Microsoft client); here is a trace from my PC:

That is completely missing from your trace - are you using the Windows built-in VPN client? Your trace starts like this:

The few RRAS events just show that RRAS event tracing was enabled.

I need a bit of time to think about the problems shown by the trace data.

Gary

Jerry 21 Reputation points

2021-06-09T19:31:36.98+00:00

Deleted.

I'm using the built-in VPN client.

What you describe pretty much matches what I see with Wireshark. Nothing goes out.
Gary Nebbett 6,201 Reputation points

2021-06-09T19:34:36.71+00:00

Hello @Jerry ,

Sorry, I answered quickly because of the attachment and made a silly error - I compared your L2TP/IPsec trace with an IKEv2 trace of mine. I just made a new trace with L2TP/IPsec and the comments about the RRAS tracing are not relevant.

Gary
Gary Nebbett 6,201 Reputation points

2021-06-09T20:05:04.42+00:00

Hello @Jerry ,

There is a lot more grovelling through the trace data to do yet, but there is something that you can perhaps answer faster - is it possible that the current best route to the remote VPN endpoint would be over another VPN interface?

Gary
Jerry 21 Reputation points

2021-06-09T20:09:11.817+00:00

Not that I'm aware of. The virtual Win10 machine hosted on the problem box connects with no issues, same network card and connection. Pathping shows 2 hops to destination
Gary Nebbett 6,201 Reputation points

2021-06-09T20:22:00.907+00:00

Hello @Jerry ,

Do you have any connection security rules defined that could possibly apply to the path to the remote VPN end-point?

This type of an error on outbound packets doesn't have many possible causes and I think that the explanation lies in this direction.

Gary
Jerry 21 Reputation points

2021-06-09T20:28:42.56+00:00

No connection security rules defined.

The connection had been working for several months when it suddenly stopped. The strongest lead I was able to find was the ISAKMP packet blocked because of clear text. That does not happen in the working virtual box. I have the trace files from both using Microsoft-Windows-WFP
Gary Nebbett 6,201 Reputation points

2021-06-09T20:40:10.547+00:00

Hello @Jerry ,

It sounds as though you are familiar with ETW. Can you try just capturing Microsoft-Windows-WFP events with stack tracing enabled and then check the call-stack on the packet drop events?

I am currently working on the assumption that this would show tcpip!IpSecTlPacketsOutProcessing as the point where the error is detected - it would be helpful to have that confirmed.

Gary
Jerry 21 Reputation points

2021-06-09T20:42:46.453+00:00

Not totally familiar but I can cope with a little help.
Has to be late tonight or tomorrow.

Answer 4

Hello @Jerry ,

It is late here already (22:50), so this is the last post of the day.

One way of creating such a trace is to save the following XML to a file (say wfp.wprp) and to issue the command wpr -start wfp.wprp!WFP -filemode to start the trace. wpr -stop why.etl stops the trace.

<?xml version="1.0" encoding="utf-8"?>  
<!-- wpr -start wfp.wprp!WFP -filemode -->  
<!-- wpr -stop ?.etl -->  
<WindowsPerformanceRecorder Version="1.0" Author="Gary">  
  <Profiles>  
    <SystemCollector Id="NTKL" Name="NT Kernel Logger">  
      <BufferSize Value="64" />  
      <Buffers Value="128" />  
    </SystemCollector>  
    <EventCollector Id="ETW" Name="ETW">  
      <BufferSize Value="64" />  
      <Buffers Value="128" />  
    </EventCollector>  
    <SystemProvider Id="Map">  
      <Keywords>  
	<Keyword Value="Loader" />   
      </Keywords>  
    </SystemProvider>  
    <EventProvider Id="WFP" Name="Microsoft-Windows-WFP" NonPagedMemory="true" Stack="true" />  
    <Profile Id="WFP.Verbose.File" Name="WFP" Description="WFP" DetailLevel="Verbose" LoggingMode="File">  
      <Collectors>  
        <SystemCollectorId Value="NTKL">  
          <SystemProviderId Value="Map" />  
        </SystemCollectorId>  
        <EventCollectorId Value="ETW">  
          <EventProviders>  
            <EventProviderId Value="WFP" />  
          </EventProviders>  
        </EventCollectorId>  
      </Collectors>  
    </Profile>  
  </Profiles>  
</WindowsPerformanceRecorder>

Gary

Jerry 21 Reputation points

2021-06-10T01:51:09.427+00:00

I have 2 files. One from the bad VPN on the host and the other from the working VPN on the virtual machine

I have some obligations until 1PM NY time. I can post after that.
Jerry 21 Reputation points

2021-06-10T18:26:46.667+00:00

host is the problem machine

https://drive.google.com/drive/folders/18BNacd7cvVYKjFTxyrqhSZuRCWG8d_1h?usp=sharing

Answer 5

Hello @Jerry ,

Thanks for that. As expected, the problem is detected in tcpip!IpSecTlPacketsOutProcessing (see stack trace below). I have spent several hours today, assuming that this was the problem, trying to reconstruct/reproduce it - without success so far (don't worry about the large symbol offsets in the stack trace - that is normal for code not on the main path after profiling driven optimization).

netio!BfeEventProviderFireNetEvent+0xcb
netio!FwpmEventProviderFireNetEvent0+0x9
tcpip!IPSecFireDiagEvent+0x105
tcpip!IpSecTlPacketsOutProcessing+0x7c196
tcpip!ProcessOutboundTransportLayerClassify+0xa43
tcpip!WfpProcessOutTransportStackIndication+0x7cb39
tcpip!IppInspectLocalDatagramsOut+0x87d
tcpip!IppSendDatagramsCommon+0x41b
tcpip!IpNlpSendDatagrams+0x40
tcpip!UdpSendMessagesOnPath+0x7ad
tcpip!UdpSendMessages+0x29c
tcpip!UdpTlProviderSendMessagesCalloutRoutine+0x15
ntoskrnl!KeExpandKernelStackAndCalloutInternal+0x78
ntoskrnl!KeExpandKernelStackAndCalloutEx+0x1d
tcpip!UdpTlProviderSendMessages+0x7b
afd!AfdFastDatagramSend+0x658
afd!AfdFastIoDeviceControl+0x1825
ntoskrnl!IopXxxControlFile+0x382
ntoskrnl!NtDeviceIoControlFile+0x56
ntoskrnl!KiSystemServiceCopyEnd+0x28
ntdll!NtDeviceIoControlFile+0x14
mswsock!MSAFD_WSPSendMsg+0x402
ikeext!IkeSendPacketEx+0x3ea
ikeext!IkeSendPacket+0x7b
ikeext!IkeConstructAndSendMMResponse+0x25b
ikeext!IkeStartMMNegotiation+0x146
ikeext!IkeBeginMMInitiator+0x2bf
ikeext!IkeProcessAcquireDispatch+0x349
ikeext!IkeHandleAcquireDispatch+0xc

I will think about this some more - it would probably be helpful to future readers if we can bottom this out - but now, at 21:30 in Switzerland, I think that I am done for today.

Gary

Jerry 21 Reputation points

2021-06-10T21:24:11.037+00:00

From my searches trying to fix the problem, it is clear there are at least a few other people experiencing the same symptoms. However, none were resolved; several just wiped and reinstalled. It was one of the unresolved cases that led me to the ISAKMP packet problem. Beyond that, I'm totally in the weeds on this one.
Gary Nebbett 6,201 Reputation points

2021-06-11T07:13:13.807+00:00

Hello @Jerry ,

It would be useful to see a dump of the state of WFP on the system. The command netsh wfp show state file=why.xml creates such a dump.

The XML data includes pre-shared keys in plaintext format. You could consider if you want to try the following: create the why.xml file, edit it to replace actual shared keys with asterisks (or whatever) and briefly review the rest of the large XML structure to make sure that you are happy with it. You could then try making the XML available via Google Drive, perhaps in such a way that potential readers have to explicitly request access.

The pre-shared keys are probably presented in two formats in the XML data: "asString" (plaintext) and "data" (hex-string of the Unicode encoding of the key). The contents of the "<data>" element (or, indeed, the whole <data> element) can be deleted.

Gary
Jerry 21 Reputation points

2021-06-12T11:08:32.2+00:00

I took a shortcut and added specific rules to allow the ports inbound and outbound. Worked instantly.

Happy to be called competent. HR offices think otherwise since I don't have a computer degree and it's never been part of my job description. However, I've had to repair them since the mid-1980s in order to get my job done. I've learned a lot.

Share via

VPN stops working. No ISAKMP packets sent.

4 additional answers

Your answer