Programmatically check for SYBSYSTEM:CONSOLE or SUBSYSTEM:WINDOWS

abhishek.m 46 Reputation points
2022-06-12T06:34:33.99+00:00

I am building my C++ project using CMake. The generated exe can either be of SUBSYSTEM:WINDOWS or SUBSYSTEM:CONSOLE.. Is there a way to validate/check for the subsystem programmatically and take appropriate action ?

C++
C++
A high-level, general-purpose programming language, created as an extension of the C programming language, that has object-oriented, generic, and functional features in addition to facilities for low-level memory manipulation.
3,856 questions
{count} vote

Accepted answer
  1. RLWA32 47,246 Reputation points
    2022-06-12T07:48:22.667+00:00

    The SUBSYTEM for an .exe file is indicated by a flag in the IMAGE_OPTIONAL_HEADER structure that is part of the PE format for an .exe file. Naturally there are two versions of the structure corresponding to 32-bit/64-bit architectures. See the Subsystem flag documented at IMAGE_OPTIONAL_HEADER32 and IMAGE_OPTIONAL_HEADER64. You can use the ImageHlp Functions or plain file system functions to retrieve the Subsystem flag from the headers contained in your target .exe file.

    0 comments No comments

2 additional answers

Sort by: Most helpful
  1. Castorix31 86,986 Reputation points
    2022-06-12T08:02:23.707+00:00

    You can do

                        PIMAGE_NT_HEADERS nth = ImageNtHeader((PVOID)GetModuleHandle(NULL));  
                        if (nth->OptionalHeader.Subsystem == IMAGE_SUBSYSTEM_WINDOWS_CUI)  
                        {  
                            // Console  
                        }  
                        else  
                        {  
                            // GUI  
                        }  
    

    ...or just test GetStdHandle

    0 comments No comments

  2. Minxin Yu 12,681 Reputation points Microsoft Vendor
    2022-06-13T03:06:42.967+00:00

    Hi, @abhishek.m

    Try to use GetStdHandle function.
    For SUBSYSTEM:WINDOWS, GetStdHandle() will return NULL handles.

    Best regards,

    Minxin Yu


    If the answer is the right solution, please click "Accept Answer" and kindly upvote it. If you have extra questions about this answer, please click "Comment".
    Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.


Your answer

Answers can be marked as Accepted Answers by the question author, which helps users to know the answer solved the author's problem.