CONTENT DEPLOYMENT – the remote server returned an error: (404) Not Found

Article
11/10/2009

This is one of a series of blogs that I plan to write on troubleshooting content deployment issues. Here’s a scenario

The destination farm is running on a Windows Server 2008 computer. When deploying to the destination, the content deployment fails with the following error:

The remote server returned an error: (404)

The first thing to remember is that any errors related to content deployment will be reported to the application event log. This is the default diagnostic logging setting for Content Deployment category. Any medium events will also be reported to the ULS logs. Therefore the first place to find out why the deployment is failing is to look at the Application event log of the export server.

In my case, here’s what I found from application event logs:

Failed to transfer files to destination server for Content Deployment job '80 - 500 Full Deployment'. Exception was: 'System.Net.WebException: The remote server returned an error: (404) Not Found. at System.Net.HttpWebRequest.GetResponse()
at Microsoft.SharePoint.Publishing.Internal.Administration.HttpDataTransfer.UploadFile(String sourceFilePath, String postToUrl, String& responseXml)
at Microsoft.SharePoint.Publishing.Administration.ContentDeploymentJob.UploadFilesToRemoteServer(Guid remoteJobId, String adminPortUrl, ArrayList dataFiles)
at Microsoft.SharePoint.Publishing.Administration.ContentDeploymentJob.DoServerToServer()'

Publishing: Content deployment job failed. Error: 'System.Net.WebException: The remote server returned an error: (404) Not Found.
at System.Net.HttpWebRequest.GetResponse()
at Microsoft.SharePoint.Publishing.Internal.Administration.HttpDataTransfer.UploadFile(String sourceFilePath, String postToUrl, String& responseXml)
at Microsoft.SharePoint.Publishing.Administration.ContentDeploymentJob.UploadFilesToRemoteServer(Guid remoteJobId, String adminPortUrl, ArrayList dataFiles)
at Microsoft.SharePoint.Publishing.Administration.ContentDeploymentJob.DoServerToServer()
at Microsoft.SharePoint.Publishing.Administration.ContentDeploymentJob.ExecuteJob()
at Microsoft.SharePoint.Publishing.Administration.ContentDeploymentJob.Run(Boolean runAsynchronously)'

The Execute method of job definition Microsoft.SharePoint.Publishing.Administration.ContentDeploymentJobDefinition (ID 70d6c4ef-bda5-4b5e-84a0-8c07e4ec3e42) threw an exception. More information is included below.

So as we can see, the Exception was - The remote server returned an error: (404) Not Found. Where do we start troubleshooting? We should start by understanding what status code 404 means and what type of operation we were in when this failure occurred. The second part of the question can be easily answered by looking at the above callstack. At the top of the callstack is HttpDataTransfer.UploadFile() method. This means were were uploading a file to the target farm when the error occurred.

Info: Content deployment transfers contents to the destination by splitting them into multiple cab files and then uploading those files to a page in the target central administration site. Thus if we open the IIS log for the target central administration website, we should see the failure and more specific information about HTTP 404 response. Here’s what I found

POST /_admin/Content+Deployment/DeploymentUpload.aspx filename=%22ExportedFiles1.cab%22&remoteJobId=%223c392e22-67d9-46cb-b6c3-225cade93479%22 81 - 404 13 0 20

The salient parts in this IIS log entry are highlighted in yellow colour. The value 13 right after 404 is a sub-status code that gives more information about the type of failure. The IIS status codes and their meanings can be found here

So from this article, we end up with: 404.13 - Content length too large

So what is the size of the file ExportedFiles1.cab that we are trying to upload to the destination? We can find this out easily by opening the path to the temporary files on the source (Export Server) as configured in the content deployment settings within Central Administration. By default, this path is %windir%\TEMP\ContentDeployment. Now there is a catch here. By default, content deployment removes the temporary files. You must first configure content deployment to keep the temporary files in the event of a failure. This setting cannot be changed from the User Interface. The correct procedure to change the setting is to use STSADM command line

STSADM.EXE -o editcontentdeploymentpath -pathname "MyPath" -keeptemporaryfiles Failure

After making this change in setting, on the next run, the temporary files will not be removed and you can easily find out the size of the file it was trying to upload by opening this folder. In my case, it turned out to be 61 MB. So why can’t I upload a file of this size? It was possible to do this in Windows Server 2003! Also, who is throwing this error? These are the next set of questions we need to find answers to.

As I just mentioned, it is very important to figure out who is throwing this error – Is it IIS or Is it SharePoint? We know this behaviour was not present in Windows 2003 and therefore a likely chance that IIS 7, that ships with Windows 2008 is a cause here. Also, a little bit of IIS domain knowledge will help you very easily conclude where the problem is. If you do not have the domain knowledge, you can still troubleshoot using the Failed Request Tracing feature of IIS to pin point where exactly the failure occurred. Failed request tracing is a different subject altogether and if you are interested to know how to use this feature, here’s the official documentation.

So for folks without IIS domain knowledge, throw in the returned error code and associated message from the KB above into a search engine of your choice and you will get several articles about what setting you need to change to fix this. I’d suggest always using official documentation. I found this.

For myself, I know from experience that IIS 7 has a change where by we limit the max size of a request. The feature implements something like what URLScan did for us in the previous versions. Here is the documentation on requestLimits on IIS 7. So reading through, I can see that this module in IIS 7 can return 404.13 status and the size of request allowed is specified by maxAllowedContentLength attribute with a default value of 30000000 bytes, which is approximately 28.6MB.

So now we have identified why the server returned this status code and the solution is to simply increase this limit. Here’s how to do that using the new command line feature of IIS 7

On the destination farm, Open a command prompt and switch to C:\Windows\System32\inetsrv folder
Execute: appcmd set config "SharePoint Central Administration v3" -section:system.webserver/security/requestFiltering -requestLimits.MaxAllowedContentLength:524288000

CONTENT DEPLOYMENT – the remote server returned an error: (404) Not Found

Additional resources