Exception when calling native function with ByVal arguments in 64 bit VBA

Question

I'm trying to make a C++/CLI dll to wrap a managed assembly so it can be called from VBA inside Excel (I specifically want to avoid using COM).

It shall only support 64 bit.

I have based my test code on the dotnet cpp-cli sample and added four additional native entry points:

extern "C" MIXEDLIBRARY_API int __stdcall CalcIntByVal(const int value)

{

return 42 + value;

}

extern "C" MIXEDLIBRARY_API double __stdcall CalcDoubleByVal(const double value)

{

return 3.14 + value;

}

extern "C" MIXEDLIBRARY_API int __stdcall CalcIntByRef(const int* value)

{

return 42 + *value;

}

extern "C" MIXEDLIBRARY_API double __stdcall CalcDoubleByRef(const double* value)

{

return 3.14 + *value;

}

I made a small Excel VBA sample that I debug from Visual Studio (use excel as debugging command or attach to process).

It first calls the two 'ByRef' methods then the 'ByVal' methods.

Here is the VBA code:

Option Explicit

Private Declare PtrSafe Function SetDllDirectoryW Lib "kernel32" (ByVal lpPathName As LongPtr) As Long

Private Declare PtrSafe Function CalcIntByVal Lib "MixedLibrary.dll" (ByVal value As Long) As Long

Private Declare PtrSafe Function CalcDoubleByVal Lib "MixedLibrary.dll" (ByVal value As Double) As Double

Private Declare PtrSafe Function CalcIntByRef Lib "MixedLibrary.dll" (ByRef value As Long) As Long

Private Declare PtrSafe Function CalcDoubleByRef Lib "MixedLibrary.dll" (ByRef value As Double) As Double

Private Sub CommandRun_Click()

Dim dllPath As String

dllPath = "MixedLibrary.dll" ' <- native x64 shim

Dim dllDir As String

dllDir = "C:\temp\\cpp-cli\bin\Debug\x64" ' <- folder with shim *and* its deps

' 1) Add dir so dependencies are found

Call SetDllDirectoryW(StrPtr(dllDir))

Dim myNumber As Long

Dim dnum As Double

dnum = 456#

myNumber = 123

' ByRef calls NOT causing exception

' Call the C++ method with int

myNumber = CalcIntByRef(myNumber)

' Call double

dnum = CalcDoubleByRef(dnum)

' ByVal call causing exception

myNumber = CalcIntByVal(myNumber)

dnum = CalcDoubleByVal(dnum)

End Sub

No issue is seen with the first two XXXByRef calls, but with the XXXByVal calls an access violation exception is seen on entry to both methods.

Exception seen on call to CalcDoubleByVal

The value of the argument is also not transferred correctly for the CalcDoubleByVal method.

The exception seems to indicate that the stack or memory has been corrupted, but I cannot find the root cause.

A full sample solution can be found here.

https://drive.google.com/file/d/1IyqTzAHGkzZAkBV0JyWlDp8cjnWL2EFP/view?usp=sharing

When debugging the top of the call stack at the time of exception is:

`kernel32.dll!IsBadStringPtrA()	Unknown`

VBE7.DLL!MereLogWinApi(struct serDllTemplate *,unsigned short,void *) Unknown

VBE7.DLL!DllFunctionCall() Unknown

So it seems to be connected with some logging inside VBE7.dll just prior to making the dll call.

From the dissassembly:

sub rsp,18h

test rdx,rdx

je IsBadStringPtrA+3Dh (07FFBDBA10B0Dh)

test rcx,rcx

je IsBadStringPtrA+45h (07FFBDBA10B15h)

mov qword ptr [rsp],rcx

lea r8,[rdx-1]

lea r8,[r8+rcx]

mov al,byte ptr [rcx]

test al,al

It fails in the last line, because what seem like it is interpreting the argument as a string.

Answer 1

Thanks for sharing the details and updated sample!
The exception occurs because of stack alignment and mixed-mode issues on x64 when calling unmanaged functions from VBA under a debugger. Here’s how to resolve it:

Root Cause

• On 64-bit Office, __stdcall is ignored; all functions use the Microsoft x64 calling convention.
• Mixed-mode (C++/CLI) DLLs can introduce alignment problems if unmanaged entry points aren’t fully isolated.
• VBA Long is always 32-bit, even in 64-bit Office, so argument size mismatch can corrupt the stack.
• Debugger attachment triggers VBE logging routines, which can misinterpret arguments.

 Solution

1. Remove __stdcall (you already did this).
2. Force unmanaged context for exported functions:

      #pragma managed(push, off)
      extern "C" __declspec(dllexport) int CalcIntByVal(int value) { return 42 + value; }
      extern "C" __declspec(dllexport) double CalcDoubleByVal(double value) { return 3.14 + value; }
      #pragma managed(pop)
   

#pragma managed(push, off)

extern "C" declspec(dllexport) int CalcIntByVal(int value) { return 42 + value; }

extern "C" declspec(dllexport) double CalcDoubleByVal(double value) { return 3.14 + value; }

#pragma managed(pop)

1. Compile the DLL as pure x64 native (or move these functions into a separate native DLL).
2. Ensure VBA declarations match types:

      Private Declare PtrSafe Function CalcIntByVal Lib "MixedLibrary.dll" (ByVal value As Long) As Long
      Private Declare Ptr
   

   Long in VBA = 32-bit → matches int in C++.
   • Double matches correctly.

Private Declare PtrSafe Function CalcIntByVal Lib "MixedLibrary.dll" (ByVal value As Long) As Long

Private Declare Ptr

1. If the crash only occurs when the debugger is attached, it’s related to VBE logging and not the function itself. Test without debugger to confirm.
2. For maximum stability, consider:
   • Wrapping calls in COM.
     • Using SafeArray or Variant for ByVal arguments.

Reference

• 64-bit VBA and API declarations.
  https://learn.microsoft.com/en-us/office/vba/library-reference/concepts/64-bit-support

Let us know if the issue persists after following these steps. I’ll be happy to assist further if needed. If the issue has been resolved, please click "Accept Answer".

Answer 2

Thanks for reaching out.
The issue occurs because of a mismatch in calling conventions and argument sizes between VBA and your C++ DLL on 64-bit Office. On x64, __stdcall is ignored, and all functions use the Microsoft x64 calling convention. Additionally, VBA Long is always 32-bit, even in 64-bit Office, which can cause stack corruption when passing ByVal arguments.

Why ByVal crashes

• VBA pushes a 32-bit value for Long, but the stack alignment expected by the x64 ABI differs.
• The crash happens because the arguments are misinterpreted during internal validation.

Workaround

1. Remove __stdcall from your C++ functions. On x64, it’s not needed:
    extern "C" __declspec(dllexport) int CalcIntByVal(int value);
   extern "C" __declspec(dllexport) double CalcDoubleByVal(double value);
2. Match VBA declarations to C++ types:
   • Private Declare PtrSafe Function CalcIntByVal Lib "MixedLibrary.dll" (ByVal value As Long) As Long
     Private Declare PtrSafe Function CalcDoubleByVal Lib "MixedLibrary.dll" (ByVal value As Double) As Double
     Private Declare PtrSafe Function CalcIntByRef Lib "MixedLibrary.dll" (ByRef value As Long) As Long
     Private Declare PtrSafe Function CalcDoubleByRef Lib "MixedLibrary.dll" (ByRef value As Double) As Double
   Long in VBA = 32-bit → matches int in C++.
   • Double matches correctly.
   1. Compile your DLL for x64 and ensure Excel is 64-bit.
   2. Keep extern "C" to avoid name mangling.

After these changes, the ByVal calls will work without exceptions.

Summary:
The root cause is the ignored __stdcall and type mismatch on x64. Removing __stdcall and ensuring correct VBA declarations resolves the issue.

Let us know if the issue persists after following these steps. I’ll be happy to assist further if needed. If the issue has been resolved, please click "Accept Answer".