If you are really wondering what this post is about, I strongly recommend reading Chris’s post “Eliminating large JS files to optimize SharePoint 2010 internet sites“ on why you don’t want to load the all .js that are used by SharePoint 2010 files for anonymous facing websites.

How this stuff works within Microsoft?

However, it took some challenges to figure this one out. I would like to give you an insight how stuff works within Microsoft on how to get something like this be officially supported yes/no.
First of all, the only team responsible for saying that something is supported or not, is the Product Group (PG).

To get such a statement the following happens:

• PFE (that’s me) researches if there is a related (kb) article to support the question of the answer, if nothing found, he escalates this to the Account Manager of the customer who then raises a ticket for a Request for Collaboration.
• Technical Lead will craft a Request for Collaboration with the PG
• RFC goes to an Escalation Engineer that maintains this communication with the PG
• PG responds with an answer, while they are extremely busy with developing the next version of SharePoint
• RFC goes back to the Escalation Engineer and he/she communicates this with the Technical Lead and the involved on-site engineer (that’s me) to see if the answer is the desired one

As you may have noticed, I (as a Premier Field Engineer) don’t have a direct line with the PG. It’s not that I can call the developer lead of the WCM piece of the platform to give me an answer to something like this or why something is implemented as it is, this is of course for good reason ;)
I just wanted to point out how many steps are involved and that such a process can take up quite the time to be finished.

The solution

Sorry to have keep you waiting, however, below is the code by utilizing Chris’s solution, to prevent the registration and downloading of the .js files.

protected override void OnPreRender(EventArgs e)

{

    if (!HttpContext.Current.User.Identity.IsAuthenticated)

    {

        this.Controls.Add(new LiteralControl("<script type='text/javascript'>SP.SOD.set_prefetch(0);</script>"));

    }

    base.OnPreRender(e);

}

There you have it, by just calling a (yet) undocumented SharePoint JSOM call, will result in SharePoint not loading and registering the .js files. This method is however a public API which you can use for this particular reason.
The only you need to make sure is that this control is near the bottom of the page and not on top of the page like Chris explains in his post.

Really?

Yes, really, this is the solution. The question is, what does it actually do?
Well, it just stops the prefetching from happening. The only time the .js files are loaded is when a control is asking for it, meaning that you really should test if everything still works on your page when implementing this and you use a control that is dependent on a SharePoint .js file. Furthermore there is maybe a performance penalty you need to pay when you are 'synchronously’ downloading that particular .js file.

Also, please make note, like Chris is also explaining in his post, it’s not just the downloading of the .js files that is the issue, it’s also the manipulating of the DOM that is being prevented which results in a page that is loaded quicker and feels ‘snappier’ as well.

As confirmed by both Chris O’Brien and Waldek Mastykarz who are (to me at least :) the WCM gurus.

When I do Risk Assessments (that’s the proactive side of my job, rather than fixing issues, I’m trying to prevent issues from happening :) at customers, certain things always popup when it comes to the configuration of SQL server and I wanted to share these things with you so hopefully this things won’t be flagged when I come along for a Risk Assessment at your place  ;) I will cover the top two SharePoint-SQL ‘issues’ I find most at customers:

1. MAXDOP
2. AUTO_UPDATE_STATISTICS & AUTO_CREATE_STATISTICS

MAXDOP

MAXDOP should be set to 1 on the SQL Server instance.. why? Because the SharePoint Product Group says so.. ! But Robin hang on.. WTH is MAXDOP? MAXDOP stands for Max Degree of Parallelism, let me quote a MSDN article which describes this setting best:

When SQL Server runs on a computer with more than one microprocessor or CPU, it detects the best degree of parallelism, that is, the number of processors employed to run a single statement, for each parallel plan execution. You can use the max degree of parallelism option to limit the number of processors to use in parallel plan execution. To enable the server to determine the maximum degree of parallelism, set this option to 0, the default value. Setting maximum degree of parallelism to 0 allows SQL Server to use all the available processors up to 64 processors. To suppress parallel plan generation, set max degree of parallelism to 1. Set the value to a number greater than 1 to restrict the maximum number of processors used by a single query execution. The maximum value for the degree of parallelism setting is controlled by the edition of SQL Server, CPU type, and operating system. If a value greater than the number of available processors is specified, the actual number of available processors is used. If the computer has only one processor, the max degree of parallelism value is ignored. Source : http://msdn.microsoft.com/en-us/library/ms181007.aspx 

Makes sense? In very short it comes down that SQL Server can utilize the amount of processors that are available to execute the queries in parallel and thus very efficiently. Now this does sound like a very cool nifty feature eh? It actually is.. however, sometimes SharePoint tries to produce some ad-hoc SQL queries where SQL Server has no clue what the most optimal path could be and thus it could lead to bad performance when it tries to utilize all the processors that are available. Furthermore, the PG has tested a lot with variable settings and came to the conclusion to NOT to use MAXDOP  is the most stable and performing way. And, when MAXDOP can be helpful, it is stated in the stored procedure itself. 

Now, why this is not done at most installations is most probably because a) it’s the default setting when installing SQL Server and b) although it is listed in TechNet article, it is just an one-liner in the section called “Set SQL Server options” at “Storage and SQL Server capacity and configuration (SharePoint Server 2010)” which you can read about here http://technet.microsoft.com/en-us/library/cc298801.aspx and c) setting MAXDOP to be 0 (or any number other 1) is actually a SQL Best Practice, hence the default value.

And here is a blog post proofing that MAXDOP set to 0 was not good idea and by setting it to 1 solved the issue (please note that this post is about SharePoint 2007) http://blogs.msdn.com/b/grahamk/archive/2009/06/15/troubleshooting-sql-server-deadlocks-and-performance-when-running-sharepoint.aspx 

AUTO_UPDATE_STATISTICS & AUTO_CREATE_STATISTICS

There is lot of confusion and discussion about these two settings that are set per database. The reason for the confusion is, that these settings are different per SharePoint version, for SharePoint 2007 it is recommended to enable AUTO_UPDATE_STATISTICS & AUTO_CREATE_STATISTICS.. However, since the release of SP2 for SharePoint 2007, the product team introduced a new timerjob called “Database statistics” which itself takes care in updating the statistics for the databases. This timerjob is also available in SharePoint 2010 therefore we recommended disabling AUTO_UPDATE_STATISTICS. When a new content database is created, the AUTO_CREATE_STATISTICS is set to be disabled which is the recommendation. It can happen, after a migration from SharePoint 2007 to 2010, that this option is still enabled and therefore should be set to disabled as well.

In short, SharePoint 2010, both should be set to be disabled. For SharePoint 2007, it is recommended to have them both enabled.

And just to be clear, it is all about “AUTO”, it does not mean that the statistics are never updated. It’s just that SQL Server itself is not responsible anymore to do the updating, instead we have SharePoint that is responsible for it.

For more detail and information, please read this post http://blogs.msdn.com/b/chunliu/archive/2011/11/17/auto-update-statistics-and-auto-create-statistics-on-or-off-for-sharepoint-2010-databases.aspx that covers all the information on the SharePoint 2010 databases side.

Recommended reading

• Database Maintenance for Microsoft SharePoint 2010 Products http://www.microsoft.com/download/en/details.aspx?id=24282 by Bill Bear and Bryan Porter

What’s one the first things you will turn on when you are troubleshooting performance problems? Right.. the awesomely Developer Dashboard! (honestly, I cannot comprehend on how troubleshooting was done in the 2003/2007 time.. respect! :)
I actually assuming that you know what it is, so I won’t go over what it actually is other than it helps you in finding out if the performance problems are:

1. in SharePoint or custom code
2. in SQL
3. on the client (if the dashboard is saying it’s done within 50ms and your browser (IE + Developer Toolbar or Firefox + Firebug) or Netmon that it is taking +150ms than its something between the WFE and the client..)

Qx events?

While being on-site this week, the first priority was to check for some performance problems they were having on one of their environments. So, first thing I asked was, if they could turn on the Developer Dashboard. When browsing to some sites I indeed noticed the slow performance and to my surprise saw some events that I’ve never seen before in the dashboard. Maybe you also haven’t (or if you were me and try searched for it and found nothing) you’re in luck now ;)

Just to set the right expectation, I’m not talking about Audi’s Q3 or Q5 or even the Q7.. I’m talking about the Q2 and Q3 events that are listed in the screenshot above, 21205ms is quite a long time to be spent in the database right? Luckily, in a strange sort of way, reproducing this was quite easy since every page request resulted in such long running database queries. Therefore, we could setup a SQL Profiler trace to see what was going on behind the scenes.. however, I found nothing in the trace that corresponded to the 20s the developer dashboard was telling me with the Q2 message.
So I fired up the ULSViewer again, this time to check more thoroughly what exactly was happening by using the correlation ID. What I found was the following:

• Last message before the 20s delay  : Current user before SqlConnection.Open: Name: DOMAIN\USERNAME SID: S-X-X-XX-XXXXXXXXX-XXXXXXXXX-XXXXXXXX-XXXXX ImpersonationLevel: None
• First message after the 20s delay : System.Data.SqlClient.SqlException: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server)
• More detailed message  : Failure checking user access permissions for User Profile Application Proxy 'User Profile Service Proxy'. Exception: System.Data.SqlClient.SqlException: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 - Could not open a connection to SQL Server)

That’s why we have such slow pages, a request is being made to a SQL Server from a environment which cannot be reached because of VLAN configuration. Since this was a QA farm, it was accidently pointing to a Production farm due to a misconfiguration in the QA farm. Problem fixed ;)

In the meantime, I got word from the one of the developers of the Developer Dashboard (that’s really one of the coolest things of working for Microsoft ;) stating that whenever a ‘Q’ message is listed it is because of the following :

Q stands for query.  Sometimes, we know that there has been a SQL query, but don’t want to store the SQL text or callstack.  For example, if you have a page with more than 50 SQL queries, that’s your problem.  Getting details on the 51^st query just isn’t worth the server processing costs.
Plus, we fall to a catch block which omits the detailed logging information if the connection fails.

Conclusion

So there you have it! Once more a proof that the developer dashboard is awesome and the answer to what the Qx messages mean ;)

Introduction

During a CritSit, I came across a very interesting problem that was related to the Search Service. Before I arrived onsite, this was the scope to solve the issue:

We will consider this case solved and ready to be closed once we provide the information on how the numbers of items crawled will affect your search web part.

Ok, so if you have some experience using search in your custom code, you are aware of the fact that the search is actually pretty well performing no matter the amount of items in the index right? I mean, it’s the thing I evangelize at customers when they want to aggregate data because it’s the way to get data from everywhere (cross sitecollection/webapplication) in your environment in a very fast way, security trimmed and all!

But….. there are some things you should think about when dealing with LARGE amount of data. When dealing with 10.000+ items as result sets, it becomes (even) more important to create efficient queries and try to avoiding using “Order By” on non optimized fields (just consider the Order by to be EVIL ;)

Testing SharePoint Search

Let’s take a look at the performance of the following query against different number of items in the index by using the following test scenario :

• An announcement List that holds an x amount of announcements, where x will be 1000, 10.000, 100.000;
  • Order by Rank query : 
    • Select Title, Rank, Size, Description, Write, Path, contentclass
      FROM Scope() where contentclass = 'STS_ListItem_Announcements'
      Order by 'Rank'
  • Order by Write query :
    Select Title, Rank, Size, Description, Write, Path, contentclass
    FROM Scope() where contentclass = 'STS_ListItem_Announcements'
    Order by 'Write'
  • No order by query :
    Select Title, Rank, Size, Description, Write, Path, contentclass
    FROM Scope() where contentclass = 'STS_ListItem_Announcements'
    

First, I’ve added 1000 items to the list and then performed each query 10 times in a row.

Then I added 9000 items more, which results in 10.000 items in one list. After the crawl and then by rerunning the same query, the following outcome was the result:

Then when adding an additional 90.000 items to have 100.000 items in the list, we get the following results

As you can see in the tables and the graphs, the search results coming back from the “Order by Rank” and “No Order by” stay relatively the same. However, there is quite a big difference in the performance when we are using another column for the ‘Order by’. The reason for this is, even though we set a RowLimit, this RowLimit is only important for the amount of results being returned. In order to give back a good search result, the entire index has to be queried and those results are ‘ordered by’ internally using the specified column. Because this column is not very optimized (it’s like doing a string compare on every row returned) it’s takes a lot of time for the results to be returned.

To prove my point, here is the graph of the Backend Query Latency, where it shows that most of the time is spent in the Property Store

So, what can we do about it? There are two basic things:

1. Refactor the Query. Most of the time, if we don’t really care about the relevance of the result, we want to order the results on DateTime like  ‘last modified’. Now, this means that we don’t actually need every record from the index that was like uploaded three years ago right? (it’s like the same as going to page 503 on the search results page of your favorite online search engine :). So we are going to narrow down the results that will be returned by appending this to the query “AND Write > '2011-1-1' AND Write < ‘2011-2-1’ “ will narrow down the results to only get the items from 1st of January 2011 and 1st of February 2011 (or ‘last month’). Likewise we can do same for something like ‘last week’ by using this “AND Write > '2011-1-1' AND Write < ‘2011-1-7’” (depending on the culture settings, the notation of the datetime is like this : Year-Month-Day')
   In our test scenario, I used the following query “SELECT Title, Rank, Write, Path, contentclass FROM Scope() where contentclass = 'STS_ListItem_Announcements' AND write = '2011-8-29 12:29:00' ORDER BY write”. This query gave me back 1000 items with an average of 180ms(!!!), quite a big performance gain right? ;) (The reason for the specific time is because I used ProcessBatchdata to add my announcement items, meaning that using only the date will still return 100.000 items)
2. Do the ordering/sorting in memory, a second option is to use the Rank for getting the results back based on their relevancy and then do ordering/sorting in memory in code like you would when sorting/ordering a custom collection or DataTable. But please note that doing this might affect the data integrity since the results were not ordered on that column/field but on the Rank column.
3. Have dedicated disks (SSD) for every Property Store Database, if you didn’t already knew, the search databases are the databases that actually require the most disk i/o throughput so it maybe worth looking into how good the disks on SQL Server are doing. A good read on the whole SAN discussion can be found at Ryan Campbell’s blogpost http://blogs.technet.com/b/rycampbe/archive/2011/08/23/virtualization-the-san-and-why-one-big-raid-5-array-is-wrong.aspx

Conclusion

So, to conclude, this is how provided the customer with the facts on how the number of  items being crawled can impact the performance of SharePoint Search if you use another column/field for doing the Order By on. The recommendation was to refactor the query I just mentioned and also to talk to the SAN team since they were claiming performance that wasn’t met when we did the testing.

Unexpected query execution failure, error code 282. Additional error information from SQL Server is included below. "The 'proc_GetTpWebMetaDataAndListMetaData' procedure attempted to return a status of NULL, which is not allowed. A status of 0 will be returned instead." Query text (if available): "{?=call proc_GetTpWebMetaDataAndListMetaData(?,'33B57EA5-A6C2-43EF-9647-7273C62A485E',?,NULL,1,?,?,6187)}"    

you most likely find blog/forum posts that are about SSRS integration. However, there is also another thing that can cause this issue to be reported in the ULS. Before we cover that, let’s take a look at the stored procedure. If you take a look at the following article http://msdn.microsoft.com/en-us/library/dd302800(PROT.13).aspx that describes what the Stored Procedure proc_GetTpWebMetaDataAndListMetaData is all about. The thing that we are interested in, is the return value, it states “The proc_GetTpWebMetaDataAndListMetaData stored procedure returns an integer return code ,which MUST be in the following table”, now in our case, for some unknown reason a NULL value is returned instead of a integer. That’s why SQL is complaining and thus we are getting the error in the ULS.

So, how do we get the error in the ULS? If you take a look at the following piece of code:

SPSite site = new SPSite("http://test");

SPWeb web = site.OpenWeb();

SPListItem firstItem = web.GetListItem("/Shared Documents/test.txt");

SPListItem secondItem = web.GetListItem("/Shared Documents/test.txt");

We get the following error in the ULS:

0x15D0    Windows SharePoint Services       Database                          6f8g    Unexpected    Unexpected query execution failure, error code 282. Additional error information from SQL Server is included below. "The 'proc_GetTpWebMetaDataAndListMetaData' procedure attempted to return a status of NULL, which is not allowed. A status of 0 will be returned instead." Query text (if available): "{?=call proc_GetTpWebMetaDataAndListMetaData(?,'33B57EA5-A6C2-43EF-9647-7273C62A485E',?,NULL,1,?,?,6187)}"

Let’s see what’s happening on the SQL Server side by running a profile trace..

exec proc_GetTpWebMetaDataAndListMetaData 'AB835D50-E8E0-43EA-9DA9-98E63240D661',NULL,N'Shared Documents/test.txt',NULL,1,-1,0x0105000000000005150000000965EBB84367A5FC5C4CD895F4010000,59

exec proc_GetListFields '33B57EA5-A6C2-43EF-9647-7273C62A485E','56FE5542-ED10-4D4B-B248-9009525BB736'

exec proc_GetTpWebMetaDataAndListMetaData 'AB835D50-E8E0-43EA-9DA9-98E63240D661','33B57EA5-A6C2-43EF-9647-7273C62A485E',N'Shared Documents/test.txt',NULL,1,0,0x0105000000000005150000000965EBB84367A5FC5C4CD895F4010000,6187

Now, unfortunately I’m not part of the product team so explaining why by calling the same method twice results in two different stored procedure calls is quite difficult. My only guess is that the first procedure set’s everything in place for any other calls that are made in the context we have.
If I run the stored procedures (which you should never do!)  this is difference between the two:

exec proc_GetTpWebMetaDataAndListMetaData 'AB835D50-E8E0-43EA-9DA9-98E63240D661',NULL,N'Shared Documents/test.txt',NULL,1,-1,0x0105000000000005150000000965EBB84367A5FC5C4CD895F4010000,59

exec proc_GetTpWebMetaDataAndListMetaData 'AB835D50-E8E0-43EA-9DA9-98E63240D661','33B57EA5-A6C2-43EF-9647-7273C62A485E',N'Shared Documents/test.txt',NULL,1,0,0x0105000000000005150000000965EBB84367A5FC5C4CD895F4010000,6187

Quite a different result right? So what happens if we modify the source code to get the context instead of creating our own..

SPWeb web = SPContext.Current.Web;

SPListItem firstItem = web.GetListItem("/Shared Documents/test.txt");

SPListItem secondItem = web.GetListItem("/Shared Documents/test.txt");

When running the profiler again, we get this stored procedures that are called:

exec proc_GetTpWebMetaDataAndListMetaData 'AB835D50-E8E0-43EA-9DA9-98E63240D661','33B57EA5-A6C2-43EF-9647-7273C62A485E',N'Shared Documents/test.txt',NULL,1,0,0x0105000000000005150000000965EBB84367A5FC5C4CD895F4010000,6187

exec proc_GetTpWebMetaDataAndListMetaData 'AB835D50-E8E0-43EA-9DA9-98E63240D661','33B57EA5-A6C2-43EF-9647-7273C62A485E',N'Shared Documents/test.txt',NULL,1,0,0x0105000000000005150000000965EBB84367A5FC5C4CD895F4010000,6187

Makes sense right? Since we already got the context, there is no need to set it, so far the theory is correct. But to my surprise now I got two errors in the ULS log..

06/10/2011 11:41:21.13     w3wp.exe (0x1C8C)                           0x1BFC    Windows SharePoint Services       Database                          6f8g    Unexpected    Unexpected query execution failure, error code 282. Additional error information from SQL Server is included below. "The 'proc_GetTpWebMetaDataAndListMetaData' procedure attempted to return a status of NULL, which is not allowed. A status of 0 will be returned instead." Query text (if available): "{?=call proc_GetTpWebMetaDataAndListMetaData(?,'33B57EA5-A6C2-43EF-9647-7273C62A485E',?,NULL,1,?,?,6187)}"    
06/10/2011 11:41:21.13     w3wp.exe (0x1C8C)                           0x1BFC    Windows SharePoint Services       Database                          6f8g    Unexpected    Unexpected query execution failure, error code 282. Additional error information from SQL Server is included below. "The 'proc_GetTpWebMetaDataAndListMetaData' procedure attempted to return a status of NULL, which is not allowed. A status of 0 will be returned instead." Query text (if available): "{?=call proc_GetTpWebMetaDataAndListMetaData(?,'33B57EA5-A6C2-43EF-9647-7273C62A485E',?,NULL,1,?,?,6187)}"    

In a way, this also makes sense, since the same stored procedure is executed twice.. so the question is, what is wrong with the values that are passed on by the stored procedure once the context has been set that SQL throws this error?
What does SQL itself say when we run the stored procedure.. ?

This means that something is returning a NULL value within the stored procedure.. So let’s take a look at http://msdn.microsoft.com/en-us/library/dd302800(v=PROT.13).aspx that explains what all the different result sets can be.. By looking at what kind of results we got from SQL, we can identify the result set that is being returned is the List Identifier one (see http://msdn.microsoft.com/en-us/library/dd303701(v=PROT.13).aspx) . To translate, the first column is the GUID of the List, the second column is the Type and the third one is the Row Identifier of the ListItem. By knowing this, we can look into the stored procedure and find out what section of the procedure is causing the NULL value to be returned.
The piece of SQL below is the one responsible of returning the result set, if you look closely you will notice the following “RETURN @ret” return statement… in my test case I took this piece of SQL and used the values that were passed on to the stored procedure like showed above and guess what..  the @ret value was NULL .. so we found the cause! :)

DECLARE @ret int

DECLARE @WebUrl nvarchar(256)

DECLARE @WebDirName nvarchar(256)

DECLARE @WebLeafName nvarchar(128)

DECLARE @UserId int

DECLARE @UrlDirName nvarchar(256)

DECLARE @UrlLeafName nvarchar(128)

DECLARE @WebIdSelected uniqueidentifier

DECLARE @ListIdSelected uniqueidentifier

DECLARE @ItemLevelSelected tinyint

DECLARE @UrlIsListItem bit

DECLARE @RootWebId uniqueidentifier

DECLARE @UserInfoListId uniqueidentifier

DECLARE @TypeSelected int

DECLARE @ItemIdSelected int

DECLARE @WebSiteId uniqueidentifier

DECLARE @WebId uniqueidentifier

DECLARE @Url nvarchar(260)

DECLARE @ListId uniqueidentifier

DECLARE @RunUrlToWebUrl bit

DECLARE @DGCacheVersion bigint

DECLARE @SystemId tSystemId

DECLARE @MetadataFlags int

SET @UserId = 0

SET @UrlIsListItem = 0

SET @RootWebId = NULL  

SET @WebSiteId  = 'AB835D50-E8E0-43EA-9DA9-98E63240D661'

SET @WebId  = '33B57EA5-A6C2-43EF-9647-7273C62A485E'

SET @Url = N'Shared Documents/test.txt'

SET @ListId = NULL

SET @RunUrlToWebUrl  = 1

SET @DGCacheVersion  = 0

SET @SystemId = 0x0105000000000005150000000965EBB84367A5FC5C4CD895F4010000

SET @MetadataFlags  = 6187

EXEC proc_SplitUrl @Url, @UrlDirName OUTPUT, @UrlLeafName OUTPUT

IF @UserId = 0

BEGIN

    SELECT

        @UserId = UserInfo.tp_Id

    FROM

        UserInfo WITH(NOLOCK)

    WHERE

        UserInfo.tp_SiteId = @WebSiteId AND

        UserInfo.tp_SystemId = @SystemId

END

SELECT TOP 1

    @ListIdSelected = ListId,

    @TypeSelected = Type,

    @WebIdSelected = WebId,

    @ItemIdSelected = DocLibRowId,

    @ItemLevelSelected = Level

FROM

    Docs WITH(NOLOCK)

WHERE

    SiteId = @WebSiteId AND

    DirName = @UrlDirName AND

    LeafName = @UrlLeafName AND

    (Level = 255 AND LTCheckoutUserId = @UserId OR

     Level <> 255 AND (LTCheckoutUserId IS NULL OR LTCheckoutUserId <> @UserId))

ORDER BY

    Level DESC

IF @WebIdSelected <> @WebId

BEGIN

    SET @ret = 1

    RETURN @ret

END

SELECT @ListIdSelected, @TypeSelected, @ItemIdSelected

IF (@ListIdSelected IS NOT NULL AND @ItemIdSelected IS NOT NULL)

    SET @UrlIsListItem = 1

So there you go, if you come across this error in your ULS then you now know what the problem might also be.. Btw, this is fixed in SP2010 where the @ret value is by default set 0 ;)

I was at a customer last week week who had some interesting questions, one of the questions was “Is it possible to modify the alerts that are sent from a discussion list, since the link is to the item itself, rather than the discussion it’s part of?”

See the above picture? I’m talking about the “View When will SP1 be released” link, by default, this points to Lists/<ListName>/DispForm.aspx?ID=<ID>. But, the discussion list is a bit different from other lists in SharePoint. For starters the list uses two content types, Discussion and Message. When you start a new discussion, you are actually creating a folder. Then each reply to that discussion is being stored in that folder.  If you look closely at the URL when displaying a discussion, there is no dispform.aspx being used to render the page but instead there is a Threaded.aspx or Flat.aspx (depending on the view that is choosen). Next, when you look even more close, it accepts a parameter called RootFolder that is a server relative URL that looks like this “RootFolder=/sites/<siteCollection>/Lists/<ListName>/<DiscussionName>” or as in my example it shows “/sites/alert/Lists/Team Discussion/When will SP1 be released”.  In order to prove that every discussion is a folder, I came up with the following piece of code:

SPList list = web.Lists.TryGetList("Team Discussion");

foreach (SPListItem dicussion in list.Folders)

{

    Literal literal = new Literal();

    literal.Text =

        "<BR>"+

        "Discussion : "+ dicussion.Name.ToString() + "<BR>" +

        "Replies : " + dicussion.Folder.ItemCount.ToString();

    this.Controls.Add(literal);

}

Which results in

And here is the actual list (to prove my point ;)

Back to the question, to show the discussion instead of the reply, we have to change the URL from Dispform.aspx?<ListItemId> to RootFolder?<Server-relativeUrl-of-discussion>.. seems pretty easy right? So, let’s get to it!

This was actually for me the first time I’ve ever looked at into modifying alerts. I knew this was possible by editing the alerttemplates.xml file but like I said, this was my first time with customizing alerts. So I went on searching for some articles and I found this blogpost from a few years ago from the Developer Documentation team called Customizing Alert Notifications and Alert Templates in Windows SharePoint Services 3.0 and while reading through, the last paragraph called my attention since it said “In my next post I'll talk about how to use the IAlertNotifyHandler interface to customize alerts.” That seemed like something was something was possible using the Object Model (yes!) instead of modifying the XML..

Essentially that approach is about three steps

1. Create a custom handler that intercepts the email.

   public class SPAlertNotification : IAlertNotifyHandler

   {

       public bool OnNotification(SPAlertHandlerParams ahp)

       {

           //Makes life easier when debugging

           #if (DEBUG)

           Debugger.Break();

           #endif

    

           string rootFolder = string.Empty;

           string viewName = "Flat.aspx"; // or "Threaded.aspx"

              

           using (SPSite site = new SPSite(ahp.siteId))

           {

               using (SPWeb web = site.OpenWeb(ahp.webId))

               {

                   //In case of add (1) or a change (2) we modify the body of the message

                   if (ahp.eventData[0].eventType == 1 || ahp.eventData[0].eventType == 2)

                   {

                       SPList list = web.Lists[ahp.a.ListID];

    

                       SPListItem item = list.GetItemByIdSelectedFields(ahp.eventData[0].itemId, "ContentType", "Url");

    

                       //Each discussion is a folder in the discussion list, the RootItem being of ContentType Dicussion

                       if (item.ContentTypeId.Parent.Equals(SPBuiltInContentTypeId.Discussion))

                           rootFolder = item.Folder.ServerRelativeUrl;

    

                       //Each reply in a discussion is a listItem in the folder, the ContentType of a reply is Message

                       if (item.ContentTypeId.Parent.Equals(SPBuiltInContentTypeId.Message))

                           rootFolder = web.GetFolder(item.Url).ParentFolder.ServerRelativeUrl;

    

                       //Instead of returning the DispForm.aspx, we are returning the url of the view (Flat.aspx)

                       //which requires the rootfolder of the discussion.

                       ahp.body = ahp.body.Replace("DispForm.aspx?ID=" + item.ID.ToString(),

                                                   viewName +"?RootFolder=" + rootFolder);

                   }

    

                   //Send the email the alert

                   SPUtility.SendEmail(web, true, false, ahp.headers["to"].ToString(), ahp.headers["subject"], ahp.body);

               }

           }

    

           return false;           

       }

   }
2. Create our own AlertTemplates.xml file, (creating a copy of the existing one and using the copy is the easiest)
• Per  type of list, you can add a reference to your custom handler. In this case, we are only interested in modifying the email that is sent for discussion lists. Let’s see how an alerttemplate looks like
  <AlertTemplate Type="List"  Name="SPAlertTemplateType.DiscussionBoard">
    <EventTypes IsVisible="True"/>
    <Format>…. </Format>
    <Properties> 
          <!--This is our custom handler being defined here –>
         <NotificationHandlerAssembly>SP.DiscussionAlert, Version=1.0.0.0, Culture=neutral, PublicKeyToken=dc834ccbbe356fc6</NotificationHandlerAssembly>
         <NotificationHandlerClassName>SP.DiscussionAlert.SPAlertNotification</NotificationHandlerClassName>
         <NotificationHandlerProperties></NotificationHandlerProperties> 
    </Properties>
    <Filters IsVisible="True"> ….</Filters>
  </AlertTemplate>
3. Register the customized alerttemplate file to the WebApplication of your choice, now this can be done using STSADM but that’s old fashioned ;) So why not register it using the OM and let the IT Pro just click on “Activate” when the solution is deployed eh? Much more fancy imo.. therefore let’s create a WebApplication-scoped FeatureReceiver like this:

   /// <summary>

   /// Adds the modified alert templates file to the webapplication

   /// </summary>

   /// <param name="properties"></param>

   public override void FeatureActivated(SPFeatureReceiverProperties properties)

   {

       SPWebApplication webApplication =

           (SPWebApplication) properties.Feature.Parent;

          

       SPAlertTemplateCollection alertTemplateCollection =

           new SPAlertTemplateCollection(

           (SPWebService)(webApplication.Parent));

    

    

       alertTemplateCollection.InitAlertTemplatesFromFile(

           SPUtility.GetGenericSetupPath("TEMPLATE\\XML") +

           "\\CustomizedAlertTemplates.xml");           

   }

    

   /// <summary>

   /// Puts the original alert templates file back to the webapplication

   /// </summary>

   /// <param name="properties"></param>

   public override void FeatureDeactivating(SPFeatureReceiverProperties properties)

   {

       SPWebApplication webApplication =

           (SPWebApplication)properties.Feature.Parent;

          

       SPAlertTemplateCollection alertTemplateCollection =

           new SPAlertTemplateCollection(

           (SPWebService)(webApplication.Parent));

    

    

       alertTemplateCollection.InitAlertTemplatesFromFile(

           SPUtility.GetGenericSetupPath("TEMPLATE\\XML")

           + \\alerttemplates.xml);

   }

So there you have it.. I hope the inner workings with discussion list is a bit more clear now. I still haven’t covered the difference between the Threaded and Flat view and how SharePoint is doing that but maybe that’s another post..

Let me know if you find the alert modification useful :)

Code analysis within MSOCAF focuses on areas like memory management, security vulnerabilities, exception management, object model usage, quality gates for unsupported features and reporting. The framework leverages existing tools like FxCop, CAT.Net, and SPDisposeCheck to analyze custom solutions.

So how does it work? Well, first you need to download the tool from https://caf.sharepoint.microsoftonline.com/Default.aspx, then start it up, you will get this screen (looks nice and shiny eh?):

Next, click on Analyze. It will tell you how the folder structure of your deployment must look like:

Click on Next to move on, there you will be told which rules will be used to check the code:

Now, probably a lot of ‘standard’ rules will not really apply to your code. I mean, personally, I’ve never used any P/Invokes (at least, not that I’m aware off ;) so the chances are unlikely that the “Move P/Invokes to NativeMethods class” rule in the Design section will be fired. So instead, focus on the rules that are defined in the custom rule set and the Memory Management one.

After reviewing the rules, click on Next to get the screen where you can locate the folder structure that holds your solution

Now, let me share that will be reviewed.. and be aware this is bad code!

Curious what the tool will be say of this? So am I!

And here we are, as a surprise, my code did not succeed in passing the rules. And the cool thing is that you can also where exactly (line number) the code was bad, and also in most cases there is a resolution you can click on.
In this case, it’s giving me the solution to fix this and also a helpful link to a blogpost every SharePoint developer should know better than the date he/she got married..

There you have it.. you might wonder what the audience is for this tool. I believe that it’s good for both developer and IT Pro.. it gives the developers a nice little framework on which they can validate their code own. On the other hand, it gives the IT Pro a better understanding if the customization that has been given to them is really that good as the developers say it is. And if something is wrong with it, they can go back to the developers and demand an explanation why they should put into production.

Let me know what you think about it!

What is PFE? It stands for Premier Field Engineer and in short, the main purpose is to provide reactive and proactive support to Microsoft customers. For more info check out https://careers.microsoft.com/

So, how is my life any different than working as SharePoint consultant/developer? Well, for starters, the duration of engagements I have with customers is brought down to a maximum of 4 to 5 days, most of the time it’s just 1 or 2 days. So that’ s quite different than the 3-6 month engagements I was used to. And actually this is one of the things I really do like, the variety amongst the customers, the traveling and the experience of dealing with different cultures in such a fast pace.

In this post I wanted to share my experiences with the infamous (https://www.youtube.com/watch?v=GIGtHhAfe8w) critsits. What does a critsit actually mean? Well, its stands for “Critical Situation” and it means that whenever you’re on-call, you will get an IM/e-mail/phone-call that you have to go as quickly as possible to the customer to solve their problem. For me it means, getting in contact with the internal travel desk to arrange a flight (if I need to fly) and hotel and then just go. Does this mean that the critsit will begin when I arrive on-site? No! Way before we even get called there is another group of engineers working on the case. They are called the  Escalation Engineers, these guys are the real heroes, since they are figuring everything out behind the scenes. Giving us, the PFE, an action plan what to do when we arrive on-site and how to solve the problem if we are lost. Also, the data we gather as PFE, is being send these guys to work on behind the scenes. So, to give you some examples, here are two experiences where I couldn’t solve the problem without them.

Critsit #1 : “Search crawler hangs”
At this one, I was send to a customer who was facing problems with their crawler. They had some dashboards to showed information that were search driven so it was mission critical for them to solve the hanging crawler because the dashboards were showing outdated information. So, first things first, I took a look at their Search Administration page and indeed, the crawler was running for 145hours and nothing was happening. Then I did what almost everyone would do in that situation, check the EventViewer and the ULS logs.. and it showed NOTHING related to the search service.. Aargh#1
So, I tried restarting the search service and see what would happen. Well, as you would have guessed, once it was running, it returned to the exact same state the crawler was in previously.. Aargh#2
At this time I was having a call with the Escalation Engineer what to do next, and he pointed me in the direction of looking at SQL. Determining if maybe there is a deadlock  between two processes. And guess what, there was a deadlock! Now I will not going into much detail here but it turned out that a stored procedure was in an infinite loop (remember, you should never ever touch the database! ;).
To fix this, we modified this particular stored procedure to able to break out of the loop. Restarted the search service, did a reset of the index, stopped the search service. Restored the stored procedure, started the search service, did a full crawl and everything was working again and the crawl completed within 1 hour.
And everyone was happy.. Now, did it feel dirty? YES! Did it feel unnatural? YES! Why? Well, I was doing something that is on the list of forbidden things to do, even when working for Microsoft, we were still very hesitant to do this action.  And it was a last resort action we had to perform..

Critsit #2 : “w3wp process crashes randomly”
When I read the problem description, I immediately thought of custom code that had some ‘dispose’ problems. So, I asked for the custom code, to look at it in my hotel room. And, unfortunately for me, the code was perfectly fine, meaning that something else was crashing the w3wp process. The question of course was.. what was it? After excluding some scenario’s and checking the ULS –, IIS logs and the EventViewer to find out if there was a correlation between all these three. According to the ULS logs, ‘nothing’ was wrong at the moment of crashing. Next thing was maybe the amount of memory being consumed what caused the crash, while the process was peaking at 1.2gb’s of memory, it crashed at 600mb. So that couldn’t be the problem.
Yet again it was time to call the escalation engineer again to help out. Now, I’m not going into a lot of detail again but it turned out to be something was corrupting the memory heap and the garbage collector was detecting this and therefore it crashed (gracefully) the w3wp process. How did we managed to get so far? Well, there is a tool called windbg (check out Download and Install Debugging Tools for Windows) , and you attach this debugger to a process. In our case the ‘faulty’ w3wp process and then set a breakpoint on which it crashed all the time. Then when it hits the breakpoint, we create a dump of the memory and then some magic happens here at Microsoft and it turned out to be another thread was doing this, this thread being the garbage collector thread.
Now to give you some background, this customer was running SharePoint 2007 with a custom implementation of EBS (the predecessor of RBS), which has some known issues but none of them were applicable here. So, we couldn’t work this out, meaning that the customer has to upgrade to the latest CU and then the Product Group is going to take a look at it.. Aargh#3 Conclusion : problem not solved yet, will keep you posted on the outcome!

Now for me, these two critsits were awesome. I mean, I only heard of the tool windbg, I did not know what a corrupt memory heap was, and now I know! Although at times, it felt like being a puppet in a way, I did learn a lot from this. And that’s the thing I wanted to get across, even after being a SharePoint developer for six years, I’m still learning SharePoint! :)