Hello @tn-57-gs ,
I wrote a blog entry on a related problem a few years ago: https://gary-nebbett.blogspot.com/2018/10/establishing-vpn-connection-from-macos.html
Since the "next" packet should come from macOS, the best place to start would be to examine the "racoon" log entries on the Mac.
From the Windows side, one could trace the IKE exchanges, using ETW (Event Tracing for Windows) and the "IKEEXT Trace Provider"; knowing the contents of the IKE_AUTH packet from Windows to Mac might give a hint about the problem. The "IKEEXT Trace Provider" is a WPP provider, so you probably won't be able to understand the trace data by yourself; if you are happy to share it then I would take a look.
Gary