Now lets see how dumb I can make myself look in 140 characters or less! 

So what’s the problem? ASP.Net Session state uses a sliding expiry which by default should expire the session state object after 60 min of not hearing from the client for which the session was created. So while this all works fine the issue is that SharePoint never cleans up these session state records out of the ASPStateTempSessions table. The result is your SSP DB(s) may continue to grow in size.

The Fix

Fortunately the fix is trivial however does involve a few manual steps which I will outline here. The goal is to create a SQL Agent timer job which will run once a minute and clean up expired sessions by executing the stored procedure ‘DeleteExpiredSessions’

1. Ensure the SQL Server Agent (SQLSERVERAGENT) service is started and set to startup when the machine starts. At a minimum this service needs to run on all SQL servers/instances which host an SSP DB.

2. Change to the .net Framework directory where ASP.Net is located, eg %windir%\Microsoft.Net\Framework[64]\v2.0.50727.

3. From the framework director run the following command replacing [YOUR SSP DB NAME] with the name of your SSP DB: aspnet_regsql -sstype c –d [YOUR SSP DB NAME] -sqlexportonly fix.sql –ssadd

4. The command from step 3 instructs the aspnet_regsql utility to generate a sql script named ‘fix.sql’ which can be used to install the ASP.Net session state stored procedures, tables, and jobs. Since everything we need for ASP.net session state is already in the SSP DB and we don’t want to edit that DB directly or risk entering into the land of “Unsupported” we only need to execute a portion of this file. Open fix.sql into SQL management studio and locate the line which has the following text:  /* Create the job to delete expired sessions */  Highlight everything below this line to the end of the file. With this text highlighted execute the query (hit F5).

5. At this point you should be able to navigate to the SQL Agent service and see the job: [YOUR SSP DB NAME]_Job_DeleteExpiredSessions. As I mentioned previously this job will execute every one min and clean up expired sessions.

6. Repeat for each additional SSPs in the farm.

To use the code you will do something like:

using (new AppPoolImpersonator())

{

//unimpresonated code here

}

The class looks like so:

public class AppPoolImpersonator : IDisposable

{

WindowsImpersonationContext _windowsImpersonationContext;

public AppPoolImpersonator()

{

if (_windowsImpersonationContext == null &&

!WindowsIdentity.GetCurrent().IsSystem)

{

_windowsImpersonationContext = WindowsIdentity.Impersonate(System.IntPtr.Zero);

}

}

public void Dispose()

{

Undo();

GC.SuppressFinalize(this);

}

public void Undo()

{

if (_windowsImpersonationContext != null)

{

_windowsImpersonationContext.Undo();

_windowsImpersonationContext = null;

}

}

}

If prepareToMove is run within a SharePoint farm that has at least the IU that Content DB will no longer participlate in profile syncronization; even if you remove and reattach the content DB to the same farm. The issue is due to a change in behavior that we introduced in the Infrastructure Update (IU). Before the IU when a content DB was removed and reintroduced back into a SharePoint farm the DB ID would change/regenerate. This had the side effect of causing all content within those DBs to undergo a Full Crawl by search. As you can imagine this a huge overheard so in the IU we ensured that the Content DBs never had to change IDs when reintroduced into a Farm. The PrepareToRemove was originally needed to ensure we did not have to redo a full sync when the same content DB with a new ID was reintroduced. Since this ID now does not change this is no longer needed when running within a farm that has at least the IU installed.

PrepareToMove sets 'Moving' to True in the SiteSynch table of the SSP DB which signals to the profile sync job to ignore the DB which has the effect of not allowing profile syncs to occur for this DB. After attaching a Content DB back into the IU farm 'Moving' is still set to True. Even after the sync timer job fires and the stored procedure profilesynch_RegisterSiteToSynch is executed this content DB will still have its 'Moving' set to True.

The fix/resolution is to run PrepareToMove again (ok so I lied there is a case where you want to run it with the IU) but this time with the –undo option. The -undo command is harmless and it will set 'Moving' back to False and allow profile syncs to this content DB to start running again.

Be on the lookout for new TechNet and KB guidance around this command.

WARNING: Screen buffer size will require 1 MB of memory per window.

Obviously the guy or gal that wrote this message has not looked at the memory consumption of Outlook 2007.:-) The message actually goes way back. I remember filing a bug, which was subsequently rejected, on this dialog when Windows 2000 was still in Beta. As far as I can tell it is a holdover from the days when the premier operating system around was known simply as NT. As best as I can figure it was put into the OS around the NT 3.x days however if anyone is still running those version and can confirm or deny I would love to know. But here it is in Windows Vista still alive and kickin.

So let's take a look at a sample. For my scenario I want URL field however I want to control how that field is rendered. By default a URL field just renders the URL without wrapping that URL in an <a> tag. In our example we want to create a true hyperlink so we are going to generate a RenderingPattern which will wrap an <a> tag around the field value. In addition we want the creator of the field to choose what happens when that link is selected by the user. That is, will the navigation happen within the same window or will the browser open the URL in a new window. I call these options "Self" and "New" respectively.

Now that we have our scenario let's take a look at what it will take to wire up the new field. First I need to define our field via fldTypes XML. Code 1 shows what this looks like. As you can see the Parent is "URL" because we want to use the same storage and rendering of the built-in URL however we are going to tweak it a bit. The FieldTypeClass will be outlined in Code 2. The PropertySchema field is used to collect the user's intent on how the URL will behave within the client's browser and finally we have RenderingPattern CAML which will control how the field will be rendered within the browser. Note I have a FieldSwitch on a Property called 'HowOpenUrl', this value will not be chosen from the PropertySchema's HowOpenUrl directly however once you take a look at Code 2 you will see how this is done. The reminder of the RenderPattern uses the FieldSwitch to determine how to render the URL. The default rendering is an <a> tag with the target set to _self so the URL will be navigated to within the same window.

Code 1 – fldtypes_sample.xml

After creating a Class Library project and adding the proper reference to Microsoft.SharePoint I added the following code to define my ConfigurableUrlField. This field derives from the SPFieldURL class which is important because we defined in Code 1 that our ParentType is "URL". Like all custom fields we need to create the two proper constructors which not only allow our type to be constructed but properly instantiates the base type. Next we need to override the OnAdded() and OnUpdated() methods which each get called whenever this field definition is added to a new site column or when the site column definition is updated, for example when someone wants to change from a rendering of this field from "Self" to "New". The magic all happens in ConfigureSchemaXml(), this is where we access the base.SchemaXml and update it with our custom property's value as configured by the user. This is the only chance we get to manipulate the schema before being stored in the DB and used to render the field. Note the name of the attribute being added/updated needs to match the name being used within the FieldSwitch: <Property Select='HowOpenUrl'/>

Code 2 – ConfigurableUrlField.cs

For completeness I have included the manifest.xml used for deployment of the solution. I will not go over that here since the concept is covered in so many other places.

Code 3 – manifest.xml

Once we have our solution added and deployed within our farm it is time to use the field. I created a test list and added a new column based on the Configurable URL field I just deployed. Note the open to set how the URL will open within the browser.

Now back at my list I create a new list item – note that we did not create a custom field control but rather are using the SPFieldUrl's field control for editing of this field.

Finally we can see the URL in action. Obviously an image does not do it justice but if I look at the HTML source I can find what actually got rendered:

<a target="_self" href="https://www.msn.com" href="https://www.msn.com">https://www.msn.com</a>

I think of downtime in two ways, 'not available' and 'reduced functionality'. The second obviously is more closely aligned to zero downtime however for a large farm it is difficult logistically to achieve.  Reduced functionality is providing your users with a read only farm during the time in which the patching of the primary farm is occurring. This requires the entire primary farm to be swung to another farm with its content DBs set to read only mode. Once patching on the primary farm is complete traffic is redirected back to the primary farm.

Since we want to always keep our downtime window as small as possible we should always follow a good and well tested practice of upgrading a farm to ensure there are no surprises along the way that will impact our downtime. There are two phases of upgrade, laying down updated binaries and running psconfig. The second phase, running psconfig, is the one that is going to take the majority of the time and the time taken is directly proportional to the amount of site collections within the farm. Psconfig upgrades each content DB schema and in many installations this can take many hours to complete. We have found in our testing in many real world deployments that detaching all content DBs other than the CA content DB, running psconfig, and then attaching content DBs back can help reduce the amount of downtime needed when patching SharePoint. One myth that needs to be dispelled now; the reduction in downtime is not achieved because psconfig runs faster or gets better throughput when upgrading the Content DB.

As with most things in SharePoint there are rules around the DB Attach process:
1. Only one Content DB can be attached to a farm at any one time.
2. Once a Content DB has been attached to a farm all of its content is marked as updated and therefore will incur what is effectively a full crawl the first time search crawls this DB. More on this later.

 So taking into account rule #1 there are two ways we can optimize this process to reduce downtime, 1) Prioritize, attach content DBs that belong to Web Applications that are highly sensitive to downtime first and make them available,  and 2)Use surrogates, build out additional worker farms which are used as surrogates to host attaching content DBs.   

 The process of using surrogate farms includes building one or more single server, throw away, farms that are running the same patch level you are upgrading your primary farm to. This process has been well documented here. This approach however has downsides such as the need for additional hardware, the additional time and effort to build out these farms, and the need for your SQL server to be able to handle the additional load.  

 Let's take a look at the steps of a typical upgrade with DB prioritization:

0. Announce and coordinate downtime with IT, users, etc.
1. Take farm offline, typically you are pulling the WFEs out of a load balancer or for the case of reduced functionality, swinging DNS settings to another read only farm replica.
2. Detach all content DBs from the farm.
3. Run WSS and if applicable MOSS upgrade patches on each server and choosing  to not run psconfig. You are only going to run psconfig once, not once for each upgrade package.
4. After all servers have been patched run psconfig on the CA machine. The execution of psconfig will not take near as long because all the content DBs are detached.
5. For each additional machine in the farm and one at a time run psconfig.
6. At this point your farm, without any content DBs, is upgraded and only the content DBs require upgrade.
7. Starting with the web application that is most critical to get back into production start attaching its content DB(s). Once complete put this Web Application back in the load balancer and notify everyone it is back online.
8.Continue running through each additional Content DB until each is attached back into the farm.

I have a tool that I will be releasing soon named CDBManager that will help with the DB prioritization method of upgrade, specifically it allows you to:

• Mass DB detach all the content DBs in a farm
• Reorder content DBs by priority
• Automate the attaching of content DBs and provides ETA of when each DB will be complete (important because there is not a progress indicator otherwise)
• Manages the upgrade.log file by creating an upgrade.log file for each content DB attached. As you may know each time a Content DB is attached to a farm and upgraded a new upgrade log is either created or if one already exists it is appended too. The problem with this approach is that all your upgrade logging is in a single file and not split out by Content DB. CDBManager renames the upgrade.log file after each Content DB has completed attaching with the name of the Content DB. This makes it much easier to go back through each log and analyze what might have gone wrong within a certain Content DB on upgrade.

I have a couple of large enterprise customers that are testing the tool now. Once we get past any breaking issues I publish the bits.

One additional point about the prioritization upgrade approach; should a content DB fail to attach along the way for whatever reason you should continue to DB Attach the remaining content DBs. While the upgrade is progressing along you now have the opportunity to investigate and mitigate the issues with the failed content DB in parallel to the upgrade and once ready retry the DB attach operation.

 So time to revisit DB Attach rule #2. When a content DB is attached into a farm its ID is changed. This has the side effect of effectively marking each object within the content DB as changed. This means that when the crawler service hits this content DB, weather doing an incremental or a full crawl, it will effectively do a full crawl as it believes all the content has changed. The Infrastructure Update (IU) changes all of this and effectively takes this rule out of play. After installing the IU the Content DB is not longer changed. This means an incremental crawl after re-attaching a Content DB is really an incremental crawl. No more full crawls after detaching and attaching content DBs, yea!

The Infrastructure Update (IU) KB article also has this blurb:

 Improvements to the time that is required to update and upgrade Windows SharePoint Services sites.

So what does this mean? Any fix before the IU psconfig updates each site collection in the farm by updating its build version to reflect the most recent value. After installing the IU we only hit site collections to update them if a schema object of the site collection needs to be updated, such as an update to the template schema. This type of update is far less frequently since hotfixes rarely do site collection schema updates (IU however does to support the new search features). The end result is that if a fix does not require a schema update psconfig does not go through each site collection and update the build number, so this will drastically reduce the amount of time necessary to perform the upgrade.

So there you have it, while we cannot upgrade a live farm we do have processes and available fixes that will move us closer to the nirvana of a zero downtime upgrade.

Happy upgrading!

The tweek involves modifying the metabase per https://support.microsoft.com/kb/332088 to turn on this additional logging at the AppPool. This KB has a command:

This turns the logging on for the DefaultAppPool however a more global approach, and one that would be useful esp. when you have more than one AppPool like us SharePoint guys is:

It goes without saying but after you set this you want to do some kind of monitoring for these events to get a since as to what is going on with your AppPools throughout the day.