How do I find full details of the registers used by Windows calls from masm64

Ian turner 66

I am trying to learn assembler using Masm64 in VS2019. I have progressed quite wel, until I reached the current stage of wanting to call windows functions from my asm files. My problem is I cannot find any for of listing of the registers that need setting up before calling say printf() and all the other functions I am used to using in C , C++ and c# .

As an example, I tried to use gets, what should be a a simple call, but it crashes immediately on return from the call with "Access violation writing location 0x00000000FFFFFFFF". I already push and pop the normal registers before starting, and have tried using sub rsp, 40h, and mov rcx, offset buffer but it continues ot crash. Clearly I need to set some register(s) but have spent 3 days searching with google to find some documentation, but with no luck whatsoever, I just get the same old YouTube videos and StackOverflow results, none of which give me the answer I need.

Where can I find the relevant Windows (10) documentation to let me call these functions form my asm code ??.

1 answer

Xiaopo Yang - MSFT 11,336 Reputation points Microsoft Vendor

2021-11-15T02:39:20.617+00:00

Have you seen x64 calling convention? You can also learn by generated Disassembly in VS studio.
Please sign in to rate this answer.
Ian turner 66 Reputation points

2021-11-15T09:19:01.86+00:00

Yes, I have been through all of that documentation, including stack etc, but you appear to be misunderstanding my question ?

My problem is that let say I want to call printf from my X64 asm code. I know I have to put the string to be printed in a register, but which one ? Equally, I know from my research to date that printf() needs various other registers setting up BEFORE the call is made. I have no idea what these REQUIRED registers are, or what data type/items they should contain, and this applies to all other API style calls to the windows libraries.

A more simple example is gets(), which "I assume "needs a buffer for the string entered to be stored into, but into what register should that buffers address be placed, and what other registers need prefiling. I have tried using gets() with and without a buffer being passed in virtually every register, but it still crashes on return from the call with error message "0xC0000005: Access violation writing location 0x00000000FFFFFFFF", and it clears ALL of the registers but RIP and RSP, so clearly I have not provided the correct setup information ?

That register name/data type info is what I am trying to find. I assume that these must all be listed somewhere, as is the case with say the Intel processor documentation, which has every instruction and function with full setup and result information.

Xiaopo Yang - MSFT 11,336 Reputation points Microsoft Vendor

2021-11-16T01:34:15.41+00:00

Integer arguments are passed in registers RCX, RDX, R8, and R9. Floating point arguments are passed in XMM0L, XMM1L, XMM2L, and XMM3L. 16-byte arguments are passed by reference. Parameter passing is described in detail in Parameter passing.

There are some Examples.

Xiaopo Yang - MSFT 11,336 Reputation points Microsoft Vendor

2021-11-16T03:01:03.187+00:00

The caller must always allocate sufficient space to store four register parameters, even if the callee doesn't take that many parameters(shadow space).

The following code is from Internet. You can also refer to Stack allocation.

extrn MessageBoxA : proc extrn ExitProcess : proc .data body db 'Hello World!', 0 capt db 'My First x64 Application', 0 .code main proc sub rsp, 28h ;8*4(aligned) + 8 xor r9d, r9d ; uType = 0 lea r8, capt ; lpCaption lea rdx, body ; lpText xor rcx, rcx ; hWnd = NULL call MessageBoxA xor ecx, ecx ; exit code = 0 call ExitProcess main endp end

Ian turner 66 Reputation points

2021-11-16T09:02:36.847+00:00

Sorry, but you are still not understanding my problem :-)) I fully understand the calling convention FROM C++ to ASM, as you have kindly described above in your reply. My issue is calling FROM ASM into the Windows API.

eg: mov r8, RCX
... other instructions
call gets
... process string returned by gets()

which of course receives the 1st argument from C++ that I want to print, or scan, or perform some other operation, no problems, I am handling most types of incoming data arguments, but now I want to use gets() FROM ASM to get a string from the user to use with the parameter I already have, my problem is how to set up the specific "preamble" required by (in this example) the 'call gets()' instruction in my ASM code. I have tried using this instruction, passing a buffer in every register, and sadly in all cases it crashes immediately on return from the call, AND also zeroes out all registers bur RSP and RBP, so gets() obviously needs a different "preamble" or set of registers that need presetting to some value or other before gets() will return the string I have entered in the console.

It is that "preamble" information I need to find for all the possible calls back into the Windows API from ASM. Hopefully that clarifies my question ??

Ian turner 66 Reputation points

2021-11-16T09:24:32.747+00:00

I just noticed your second response with asm code and have questions ?
You show a call to Messagebox, and 1st reserve shadow space for 4 x 8byte arguments, and then put various data into the standard rcx,rdx,r8,r9 registers, but my question is :

sub rsp, 28h ;8*4(aligned) + 8
xor r9d, r9d ; uType = 0
lea r8, capt ; lpCaption
lea rdx, body ; lpText
xor rcx, rcx ; hWnd = NULL
call MessageBoxA

How do I find out what of those registers take what data items/type before the call is made. In other words, how do I find out that the caption needs to be in R8 and the text of the message goes in RDX and not the other way around, or in a different pair of registers ? Your example show the sequence of arguments is HWND, text string, Caption text, Icon type, but how can I find that order out ?

Does that make sense ?

Xiaopo Yang - MSFT 11,336 Reputation points Microsoft Vendor

2021-11-17T01:45:27.337+00:00

By default, the x64 calling convention passes the first four arguments to a function in registers. The registers used for these arguments depend on the position and type of the argument. Remaining arguments get pushed on the stack in right-to-left order.
Sign in to comment