Inline Assembly Call to compiled Subroutine gives access violation executing location

Question

I have recently updated a large C++ program from Visual Studio 2005 to Visual Studio 2019. All has worked well except for this one problem where using inline assembly, we call into a compiled subroutine. Stepping into the "CALL" in disassembly shows that we are accessing the correct location, yet when the first instruction, "push ebp" is executed, I get an Access violation executing location 0x07202240 for example. I have confirmed the parameters are being pushed on the stack correctly, and that the address is correct for this method. I have also gone back to VS 2005 and stepped through the whole process watching memory and comparing to the memory in Visual Studio 2019 and confirmed that it is being done identically, yet in 2005 it succeeds and in 2019 it gives me the access violation. Any thoughts or ideas?

I also create an x86 VM running an x86 version of Windows and ensure the machine settings in Visual Studio are to target an x86 machine, so I know this is not an x64 issue with inline assembly.

Here is the inline assembly:

push ds

  mov     eCX, nbyt     ; get number of argument bytes
  sub     eSP, eCX      ; allocate room on the stack
  mov     eSI, ap       ; get argument source
  mov     eDI, eSP      ; get base of stack
  shr     eCX, 2        ; make it 4 byte words
  rep     movsd         ; copy arguments to stack
  mov     eAX           , npar; get NPAR
  push    eAX           ; save it

  mov     eDI, icp      ; get invocation context pointer
  add     eDI, 176      ; point to base of data
  ; *****************The user MUST save eBP if it is used****************
  ; *****************The user MUST save eBP if it is used****************
  call dword ptr csub          ; call the C routine
  ; *****************The user MUST save eBP if it is used****************
  ; *****************The user MUST save eBP if it is used****************

  pop     eBX           ; discard NPAR value
  mov     eCX, nbyt     ; get number of argument bytes
  add     eSP, eCX      ; remove from stack

  pop     ds            ; restore segment registers

Here is the disassembly of the function being called:

07202240 push ebp ; This is the line it gets the access violation on.
07202241 mov ebp,esp
07202243 push 72023E0h
07202248 push dword ptr [ebp+10h]
0720224B push dword ptr [ebp+0Ch]
0720224E call 072022A0
07202253 add esp,0Ch
07202256 leave
07202257 ret

Answer 1

@Viorel , thanks for your suggestion! Further investigation showed that the execute permission was not set on the memory allocated for this application. It looks like older versions of C++ allowed execution in memory allocated using malloc, but newer versions of C++ do not. I changed malloc to VirtualAlloc and set the protection level to PAGE_EXECUTE_READWRITE and my program now works correctly. Thanks for the tip!

Answer 2

Hi Viorel, thanks for the replay. Here is the whole inline assembly method. When I step through the code, I look at the memory location pointed to by eDI and confirm that after the rep movsd, the data ap which is pointed at by eSI has been copied to the eDI memory location correctly. This is what is so frustrating about this issue, is that the pointers and memory are all setup correctly at the time I "Call csub" and I step into the correct location as well. It is when doing the push ebp that I get the access violation.

int call_asm(
DWORD csub, DWORD ap, DWORD nbyt, DWORD npar)
{
__asm
{
push eBX ; save register variable
push eSI
push eDI

  push    es          ; save segment registers
  push    ds

  mov     eCX, nbyt     ; get number of argument bytes
  sub     eSP, eCX      ; allocate room on the stack
  mov     eSI, ap       ; get argument source
  mov     eDI, eSP      ; get base of stack
  shr     eCX, 2        ; make it 4 byte words
  rep     movsd         ; copy arguments to stack
  mov     eAX           , npar; get NPAR
  push    eAX           ; save it

  mov     eDI, icp      ; get invocation context pointer
  add     eDI, 176      ; point to base of data
  ; *****************The user MUST save eBP if it is used****************
  ; *****************The user MUST save eBP if it is used****************
  call  csub          ; call the C routine
  ; *****************The user MUST save eBP if it is used****************
  ; *****************The user MUST save eBP if it is used****************

  pop     eBX           ; discard NPAR value
  mov     eCX, nbyt     ; get number of argument bytes
  add     eSP, eCX      ; remove from stack

  pop     ds            ; restore segment registers
  pop     es

  pop     eDI        ; restore general registers
  pop     eSI
  pop     eBX        ; restore register variables

}
/* return value is alreay in eAX /
} / end call_asm() */