@TonEpskamp-0986 I took an existing .chm file (Microsoft Office Developer Documentation) and opened it with HtmlHelp() from a 32-bit and 64-bit minimal Windows Desktop application. The function completed successfully and opened the help file in all cases.
The documentation for the HtmlHelp function contains a caution that the executable must have a stack size greater than 100K. I changed the default stack size to 64K for my test application and then calling the function resulted in crashes for both 32-bit and 64-bit code.
I wonder if your 64-bit executable has been linked with an insufficient stack size (the default is 1 MB).
For a C++ project the project property page where the stack size can be set is shown below -
You can check the stack setting for your executable by opening a Visual Studio Developer Command prompt and using the command --
dumpbin /headers YourProgram.exe
The following demonstrates what dumpbin shows as the stack sizes for the default settings -
The figures are in hex -- the stack reserve is 1 MB and the commit is 4K.