Sharepoint – custom solution deployment failure

I recently ran into a problem where custom SharePoint solutions will randomly fail to deploy on any of the web servers in the farm. The nature of the problem was hard to debug. The failure can occur on any web server and on any resource file being deployed or when the solution is being retracted. The error message would be as follows:

"Copying of this file failed. This operation uses the SharePoint Administration service (spadmin), which could not be contacted. If the service is stopped or disabled, start it and try the operation again."

So it turns out that the internal exception that causes this problem is:

Failed to connect to an IPC Port: The system cannot find the file specified. 

There is a new hotfix available for this which updates a .NET Framework DLL, which resolves this issue. The specific KB article that talks about this is

A System.Runtime.Remoting.RemotingException exception is thrown when you deploy a SharePoint solution on a SharePoint Web server that is running the .NET Framework 2.0 SP2

FYI: You need to have .NET Framework 2.0 Service Pack 2 installed before applying this fix.

If you want to debug this and confirm that the internal exception is actually the one noted above, there are 2 ways to use this.

Easy way: Look at the application event logs to see if this exception is logged.

Hard way: !!! Do not try this in Production !!! Read on…

  1. Download and install Debugging tools for Windows for the platform on which you are running.
  2. Using ADPlus, log all first chance .NET CLR exceptions.
  3. Note down the IIS worker process (w3wp.exe) PID corresponding to your app using using task manager
  4. Save the below script to Debugging tools for windows folder as LogExceptions.cfg
  5. Execute the command as: CScript adplus.vbs –p <PID> –c LogException.cfg
  6. Reproduce the problem.
  7. Inside the debugging tools for Windows folder there is another folder that is created that begins with Crash…The exceptions are logged in a text file that is named as PID-<PID>.exe…

Here is the script:

<ADPlus>
<Settings>
<RunMode>Crash</RunMode>
<Option>Quiet</Option>
</Settings>

<PreCommands>
<Cmd>.loadby sos mscorwks</Cmd>
</PreCommands>

<Exceptions>
<Option>NoDumpOnFirstChance</Option>
<Option>NoDumpOnSecondChance</Option>
<Config>
<Code>clr</Code>
<Actions1>Log;Stack</Actions1>
<CustomActions1>.time;!pe-nested;!clrstack;!threads</CustomActions1>
<ReturnAction1>gn</ReturnAction1>
</Config>
</Exceptions>
</ADPlus>