The SUBSYTEM for an .exe file is indicated by a flag in the IMAGE_OPTIONAL_HEADER structure that is part of the PE format for an .exe file. Naturally there are two versions of the structure corresponding to 32-bit/64-bit architectures. See the Subsystem flag documented at IMAGE_OPTIONAL_HEADER32 and IMAGE_OPTIONAL_HEADER64. You can use the ImageHlp Functions or plain file system functions to retrieve the Subsystem flag from the headers contained in your target .exe file.
Programmatically check for SYBSYSTEM:CONSOLE or SUBSYSTEM:WINDOWS
I am building my C++ project using CMake. The generated exe can either be of SUBSYSTEM:WINDOWS or SUBSYSTEM:CONSOLE.. Is there a way to validate/check for the subsystem programmatically and take appropriate action ?
2 additional answers
Sort by: Most helpful
-
Castorix31 86,986 Reputation points
2022-06-12T08:02:23.707+00:00 You can do
PIMAGE_NT_HEADERS nth = ImageNtHeader((PVOID)GetModuleHandle(NULL)); if (nth->OptionalHeader.Subsystem == IMAGE_SUBSYSTEM_WINDOWS_CUI) { // Console } else { // GUI }
...or just test GetStdHandle
-
Minxin Yu 12,681 Reputation points Microsoft Vendor
2022-06-13T03:06:42.967+00:00 Hi, @abhishek.m
Try to use GetStdHandle function.
For SUBSYSTEM:WINDOWS, GetStdHandle() will return NULL handles.Best regards,
Minxin Yu
If the answer is the right solution, please click "Accept Answer" and kindly upvote it. If you have extra questions about this answer, please click "Comment".
Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.