You cannot reference both x86 and x64 DLLs in the same process as only the DLL that shares the bitness can be loaded.
Your specific issue is that you want to deploy binaries that may or may not be used in the app. To do that you'll need to add a post-build event to copy the DLLs that you need. This is pretty common actually especially when you're using something like DllImport
.