If you like to research the idea of self-modifying code, then try an experiment for Visual Studio 2019:
/*
Configuration:
- Debug, Win32
Compiler options:
- "Additional options": /Gh
Linker options:
- "Enable Incremental Linking": NO
*/
#include <Windows.h>
#include <iostream>
#include <cassert>
void f1()
{
printf( "f1\n" );
}
void __stdcall MyHook( unsigned char* retAddress )
{
//return; // uncomment to disable self-modification
unsigned char* address_of_call_penter = retAddress - 5;
unsigned char code = address_of_call_penter[0];
assert( code = 0xE8 );
unsigned char* address_of_f1 = (unsigned char*)&f1;
if( address_of_call_penter != address_of_f1 )
{
// remove 'call _penter', replace with 'nop'
MEMORY_BASIC_INFORMATION mbi;
BOOL b;
b = VirtualQuery( address_of_call_penter, &mbi, sizeof( mbi ) );
DWORD old_protect;
b = VirtualProtect( mbi.BaseAddress, mbi.RegionSize, PAGE_EXECUTE_READWRITE, &old_protect );
*(unsigned*)address_of_call_penter = 0x90909090;
( (char*)address_of_call_penter )[4] = 0x90;
b = VirtualProtect( mbi.BaseAddress, mbi.RegionSize, old_protect, &old_protect );
}
}
bool recursive_call = false;
int counter = 0;
extern "C" void __declspec( naked ) __cdecl _penter( void )
{
_asm
{
pushad
}
if( !recursive_call )
{
recursive_call = true;
printf( "_penter %i\n", ++counter );
__asm
{
mov eax, [esp + ( 8 ) * 4]
push eax
call MyHook
}
recursive_call = false;
}
_asm
{
popad
ret
}
}
int main()
{
printf( "Calling 'f1'\n" );
f1();
printf( "Returned from 'f1'\n" );
return 0;
}
The mandatory Project Properties are shown in the comment.
It includes a sample “f1” function. If _penter is called for other function, like printf, then the call is removed (replaced with ‘nop’ instructions). The next time printf will not invoke _penter. It is possible to deal with more than just one “f1” function.
The program displays “_penter” five times. When “//return” is uncommented to disable the self-modification, then _penter is called 12 times.
To detect undesirable recursive infinite calls in single-threaded environment it uses a simple flag; see also: https://learn.microsoft.com/en-us/answers/questions/123941/gh-causing-infinite-loop.html.
Works with STL functions too.