Why is ShellExecuteEx failing with SE_ERR_NOASSOC for an EXE file?

Randy P 1

I have a C application that does the following on Windows 2016 Server...

It creates a new folder
It copies an EXE file to the new folder (this copied EXE is also a C application)
It runs the copied EXE using ShellExecuteEx

This process works 95% of the time but 5% of the time ShellExecuteEx fails with SE_ERR_NOASSOC (#31).

The application performs a few retries with a 30 second delay between each retry but ShellExecuteEx still fails with SE_ERR_NOASSOC.

Only after a FEW MINUTES does ShellExecuteEx work ok and run the EXE.

I have checked on-line for ShellExecuteEx and SE_ERR_NOASSOC issues but there is nothing related to EXE files.

I do not understand how it is even possible to get an SE_ERR_NOASSOC (ie. file association error) on an EXE file since it really does not have or need an association.

I thought that maybe Windows Defender was temporarily quarantining the recently copied EXE file until it has been scanned but I would expect a different error such as ERROR_FILE_NOT_FOUND or SE_ERR_ACCESSDENIED, but certainly note SE_ERR_NOASSOC.

Any assistance debugging this strange issue would be appreciated!

Viorel 112.1K Reputation points

2022-05-07T09:57:12.977+00:00

Are you using a regular destination disk and folder?

Did you try starting the program manually when your code gives the error?

The programs can be also started using CreateProcess.
Randy P 1 Reputation point

2022-05-07T13:07:56.03+00:00

Hi Viorel,

Thank you for your comments!

I am using a managed disk on Azure.

Yes, running the program manually (many minutes after the error) works - But so does the unattended method a well. Again, this work 95% of the time.

I am using ShellExecuteEx because I need to run with the "runas" verb which is more complicated to achieve with CreateProcess.

Randy.
RLWA32 40,286 Reputation points

2022-05-07T14:08:13.09+00:00

How are you retrieving the error code 31? Was it returned by GetLastError?
Randy P 1 Reputation point

2022-05-07T14:56:58.727+00:00

...
ShExecInfo.fMask = SEE_MASK_NOCLOSEPROCESS; // want hProcess
...
if (ShellExecuteEx(&ShExecInfo) && ShExecInfo.hProcess)
{
<success here 95% of the time>
}
else
{
<ShExecInfo.hInstApp == SE_ERR_NOASSOC here 5% of the time>
}

I don't have the value from GetLastError.
RLWA32 40,286 Reputation points

2022-05-07T15:41:10.54+00:00

Try calling GetLastError also upon failure and let us know what it returns.
Randy P 1 Reputation point

2022-05-07T16:48:14.533+00:00

Will do - It will probably take a few weeks before it happens again...
RLWA32 40,286 Reputation points

2022-05-08T13:16:56.77+00:00

Have you initialized COM before calling ShellexecuteEx?
Randy P 1 Reputation point

2022-05-08T19:29:12.317+00:00
Nope, but...

The C application that is being run does not use COM

It works 95% of the time so I would expect it to fail 100% of the time if COM was needed

If COM was needed I would expect a different error message

Nevertheless, once it happens again and I get the GetLastError() value (and assuming that information does not help solve the problem), I will call CoInitializeEx on application startup.
RLWA32 40,286 Reputation points

2022-05-08T19:57:01.093+00:00

Microsoft documentation for ShellExecuteEx says "There are instances where ShellExecuteEx does not use one of these types of Shell extension and those instances would not require COM to be initialized at all. Nonetheless, it is good practice to always initialize COM before using this function."

It may not help but it can't hurt.