Why Does IInitializeWithFile Receive Multiple Callbacks in Windows 11, Leading to Explorer Crashes?

Question

Problem Statement

Overview:

• I have developed a Shell Extension to hook certain Win32 API functions in Windows File Explorer to customize its functionality. We are hooking into the CoCreateInstance COM API to intercept the creation of Microsoft's default PDF file previewer. When the PDF file previewer COM component is instantiated, we obtain the IInitializeWithFile interface pointer and override its Initialize() method by modifying the VTable to our custom implementation. As a result, Windows File Explorer launches the Microsoft PDF file previewer, causing the prevhost.exe process to load.

IInitializeWithFile::Initialize

HRESULT Initialize( [in] LPCWSTR pszFilePath, [in] DWORD grfMode );

Observations:

• Windows 10 Behavior

1. In Windows 10, the custom method invoke from shell32.dll with correct file path parameter and the call comes only once to our custom method.
2. After the custom method completed, we call the original method Initialize()

• Windows 11 Behavior

1. The custom method invoke from combase.dll in the first time, and the file path contains garbage values until the second callback iteration.
2. On the third callback iteration, the call comes from shell32.dll. However, even after successfully calling the original method Initialize() (step 2 of Windows 10),

the callbacks do not stop and continue invoking the custom function, ultimately leading to a stack overflow and crashing File Explorer.

3. Request you to share why in Windows 11 the calls comes multiple times. Where as in Windows 10 the call to our custom method comes once.

To mitigate the issue, I attempted to hook my custom method to the left mouse button click.

This approach modifies Vtable[3] dynamically, executes the logic, and then restores the original function pointer.

This method works until I click on file types such as .txt and .cpp, which do not have a preview handler.

After clicking on these files, the COM object responsible for handling the preview disappears, causing all method calls to return NULL.

Findings:

Preview Handler Process (prevhost.exe):

• When the preview option is enabled and a file is clicked, prevhost.exe starts running.
• This is a Windows system process that manages file previews in File Explorer by allowing preview handlers to display file contents (e.g., PDFs, images, documents) in the preview pane without requiring the full application to open.

prevhost.exe process Termination Issue:

• If prevhost.exe is manually terminated from task manager and a file is clicked in File Explorer again, the process prevhost.exe does not restart, preventing files from being previewed.

Request for Assistance:

• How can this behavior in Windows 11 be handled? Specifically:
• Why do the callbacks from combase.dll not stop even after sending a success response?
• Is there a way to prevent stack overflow while ensuring the file preview feature continues functioning correctly?
• How can I ensure prevhost.exe restarts automatically when required?

Any insights into what has changed in Windows 11 regarding callback behavior would be greatly appreciated.

Answer 1

Hello,

Thank you for posting in Microsoft Q&A.

Based on the description, I understand your question is related to IInitializeWithFile Receive Multiple Callbacks in Windows 11, Leading to Explorer Crashes.

In Windows 11, the behavior of the CoCreateInstance COM API has changed compared to Windows 10. Specifically, the callbacks from combase.dll are invoked multiple times, which is not the case in Windows 10. This could be due to changes in how COM objects are managed and instantiated in Windows 11. The repeated callbacks with garbage values in the file path parameter suggest that there might be an issue with how the COM object is being initialized or how the VTable is being modified.

The changes in callback behavior in Windows 11 could be related to improvements or modifications in the COM infrastructure. These changes might be aimed at improving performance, security, or compatibility with newer applications and services. Understanding the exact nature of these changes might require in-depth analysis of the Windows 11 COM API documentation and potentially reaching out to Microsoft support for more detailed information.

Here is some documents for your reference:

https://learn.microsoft.com/en-us/windows/win32/api/combaseapi/nf-combaseapi-cocreateinstance

https://learn.microsoft.com/en-us/windows/win32/api/propsys/nn-propsys-iinitializewithfile

Have a nice day.

Best Regards,

Molly

============================================

If the Answer is helpful, please click "Accept Answer" and upvote it