How are Frame Pointers used exactly when space is dynamically allocated on the stack using x64 calling convention

Philipp Barthel 1 Reputation point
2022-11-04T14:04:15.407+00:00

I am trying to reconstruct callstacks for windows x64 binaries using virtual machine introspection, meaning I can access all memory of the machine and the targeted process.
I want to accomplish this by using the unwind information that's saved by the compiler and I've come pretty far as of yet. The only piece that's missing is how frame pointers(FP) are used when space is allocated dynamically. My understanding is that the unwind code UWOP_SET_FPREG (opcode 3) appears if a FP is set to either point in the middle of allocated space in order to use shorter instructions (this case should be irrelevant for me as it means that RSP is not moved) or to denote the end of dynamically allocated space on the stack. The latter is done to reconstruct RSP in case an exception appears and thus crucial for my project as I need to know the delta RSP between return addresses(RET) of adjacent stack frames.

I have observed dynamically allocated stack space in 2 to 3 cases so far. I crosschecked my findings using WinDbg and found that the value of the register that's used as a FP also lays on the stack. It was found at R8 home twice and once at RDX home within the shadowstore of the function. These values were stack pointers and the distance between these stack pointers and the next return address were always very close to what was used by the proceeding allocate unwind codes:

  • Case 1
    used unwind codes that move RSP:
    5x UWOP_PUSH_NONVOL, UWOP_ALLOC_LARGE(18 stack locations), UWOP_SET_FPREG

distance between FPREG value and next RET:
13 locations

  • Case 2
    used unwind codes that move RSP:
    3x UWOP_PUSH_NONVOL, UWOP_ALLOC_SMALL(12 stack locations), UWOP_SET_FPREG

distance between FPREG value and next RET:
9 locations

  • Case 3
    used unwind codes that move RSP:
    8x UWOP_PUSH_NONVOL, UWOP_ALLOC_LARGE(63 stack locations), UWOP_SET_FPREG

distance between FPREG value and next RET:
61 locations

In detail my questions are:

  1. How can I differenciate if a FP is set to denote the end of dynamically allocated space or simply to use shorter instructions within statically allocated space?
  2. Why do I find the value of the FP register at different locations (R8 home vs RDX home)? How can I make sure my code will be picking the correct one?
  3. How do I properly calculate the delta RSP from the FP to the next RET? It did look like I could simply calculate it as allocate_size_in_locations - amount_of_pushes but this fails for case 3.

Thanks in advance,
Philipp

PS: I'm currently working on reconstructing the stack of a win7 SP1 notepad.exe with a breakpoint at ntdll!ZwWriteFile. Cases 1 through 3 are the functions with RET *8687, *0a03 and *1521 respectively.

Windows
Windows
A family of Microsoft operating systems that run across personal computers, tablets, laptops, phones, internet of things devices, self-contained mixed reality headsets, large collaboration screens, and other devices.
4,740 questions
0 comments No comments
{count} votes