Some of the questions included:

• How can one use SharePoint, Office 365, or other technology to apply legal preservation?
• Do you recommend or work with any particular 3rd party suppliers to add capability to SharePoint records management?
• Can SharePoint really do records management?

He published an article with the results.

16 Questions and Answers About SharePoint and Office 365 

On the document set content type there is a property called NewFormUrl, changing this property will let you specify what new form the content type uses. This allows you to specify a custom new form per document set content type. But before changing this property, a new form must be created. A new aspx page must be created and it can be stored in _layouts, or in a folder in _layouts. If you store the form inside a folder you need to add a web.config file to include the document set assembly, which is required to use the document set object model needed to create a document set.

Here is an example of a basic customized new document set page.

CustomNewDocSet.aspx
<%@ Page Language="C#" AutoEventWireup="true"  CodeFile="CustomNewDocset.aspx.cs" Inherits="_Default" %>

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml">
<head runat="server">
    <title></title>
</head>
<body>
    <asp:Literal ID="litList" runat="server" />
    <asp:Literal ID="litCt" runat="server" />
</body>
</html>

And this is an example of the code behind page.

CustomNewDocSet.aspx.cs
using System;
using System.Collections.Generic;
using System.Web;
using System.Web.UI;
using System.Web.UI.WebControls;
using Microsoft.SharePoint;

public partial class _Default : System.Web.UI.Page
{
    protected void Page_Load(object sender, EventArgs e)
    {
  SPList list = SPContext.Current.List;
  litList.Text = list.Title;
  SPContentTypeId ctid = new SPContentTypeId(Request.QueryString.GetValues("ContentTypeId")[0]);
  SPContentType ct = list.ContentTypes[ctid];
  litCt.Text = ct.Name;
    }
}

This is the configuration file that must be placed in the folder that contains the custom new document set aspx page. The config file is needed, otherwise the document set object model cannot be accessed. References can also be added other assemblies as needed. If the custom new document set page is placed in the root of _layouts rather than in a folder then this web.config file is not needed.

Web.Config
<?xml version="1.0"?>
<!--
    Note: As an alternative to hand editing this file you can use the web admin tool to configure settings for your application. Use the Website->Asp.Net Configuration option in Visual Studio. A full list of settings and comments can be found in machine.config.comments usually located in
    \Windows\Microsoft.Net\Framework\v2.x\Config
-->
<configuration>
 <system.web>
  <!--
            Set compilation debug="true" to insert debugging
            symbols into the compiled page. Because this
            affects performance, set this value to true only
            during development.
        -->
  <compilation debug="false">
   <assemblies>
    <add assembly="Microsoft.Office.DocumentManagement, Version=14.0.0.0, Culture=neutral, PublicKeyToken=94DE0004B6E3FCC5"/>
   </assemblies>
  </compilation
 </system.web>
</configuration>

In order to use the custom new document set page the URL for the document set content type must be updated. This is an example Powershell script that can be used. Pass in the URL to the site that contains the document set content type to be changed. The appropriate content type ID of the content type to be changed should be specified for ctid. This will then update the property NewFormUrl to use the customized new document set page rather than the out of box one.   $siteurl should be the URL that has the custom document set content type. $ctid is the content type ID of the document set that is being modified. $contentType.NewFormUrl is the location in relation to _layouts that contains the custom new document set form.

ChangeNewDocSetPage.ps1
$siteUrl = "http://mysite"
$ctid = "0x0120D520"
param($siteUrl)
$site = New-Object Microsoft.SharePoint.SPSite($siteUrl)
$web = $site.RootWeb;
$contentTypeId = New-Object Microsoft.Sharepoint.SPContentTypeId($ctid)
$contentType = $web.ContentTypes[$contentTypeId]
$contentType.NewFormUrl = "_layouts/CustomNewDocset/CustomNewDocset.aspx"
$contentType.Update()
$web.Dispose()
$site.Dispose()

Now a custom new document set page has been created and the newFormUrl has been changed to use the new page. When you use the New Document drop down in a document library to create a new document set you should now get your custom page. 

Thanks for reading.

Quentin Christensen, Program Manager

Carlos David Argott Hernandez, Developer, Microsoft

But what if you are in your own version of Groundhog Day? Each day doing the same things, people repeating the same mistakes, the usage of your solution is slightly increasing but not at the pace it should be. Why can’t your users see the power of productivity you provided? Why are people asking questions like we have SharePoint? What is that?

Groundhog: SharePoint makes working with other people that easy!?

Getting hundreds or thousands of people using a new thing is no small task. It takes a while to learn new things and people are slow to change their ways. Focus on adoption. SharePoint is viral, but you need to get a critical mass of people that work together to start using it, then the rest of the team will follow. However in most organizations you have teams that mostly collaborate only with each other, so you need to identify these teams and get that critical mass.

Set goals and measure adoption

If you don’t know many people are actively using SharePoint, then you don’t know if your adoption promotion efforts are successful. First you need to have goals for adoption and measurable metrics to track your success. SharePoint has out of box analytics capabilities. A basic starting point is the number of hits and number of unique users for a site collection, you can see this in the site collection usage report by going to site collection settings and selecting Popularity and Search Reports. You can also view this report for a particular Site by going to Popularity Trends in Site Settings.

Here is an example usage report. Notice this was before any adoption efforts were put in place. It’s great to have a baseline starting point so you can compare the change over time.

Here is some more information:

View and download reports about service usage in Office 365

View usage and search reports

There are also more sophisticated analytics options you can integrate with SharePoint.

SharePoint 2013: How to Choose Between Web Analytics and Google Analytics

Webtrends and everything you need to know about SharePoint adoption

Keep it simple and easy

First deploy and advertise the basic capabilities of SharePoint that need little setup and no customization. For example first get people using OneDrive for Business. Train them to use OneDrive for Business instead of their local computer to store files. Then look at deploying basic un-customized team sites that anyone in the organization can provision as needed. The faster you offer the option to people, the quicker you will realize the benefits, even if it isn’t the highly customized repository of cat images and videos you have been dying to build for your company.

Setup an Intranet portal with search

People occasionally need to access corporate resources such as new employee information, W2s, and employment information. If you can setup a single portal that is useful to everyone and everyone knows it exists, people will use it. Since you are linking people to important tasks they will use it, but ensure the experience is simple and easy. Tie that in with SharePoint search so when people can’t browse to what they are looking for, they can just search. They will start to see they can find useful content in documents and team sites and want to figure out how they can make their content available too. They will run searches and see people suggestions, click on those people and see the OneDrive for Business sites, then look into getting their own OneDrive for Business setup.  

Give them a carrot and advertise

Create one page info documents to explain how to do basic things and why. For example in a short article highlight the value of coauthoring instead of emailing attachments around. Show people the carrots that will make their work life easier, then they will be motivated to learn and use it. Create simple and easy training documents and videos and share them with people. Use email promotions, a newsletter of SharePoint tips & tricks, and articles on the company intranet home page to promote using SharePoint. 

Go viral by getting an advocate from each team

Select advocates from the teams that are using your SharePoint deployment and teach them about the benefits so they can spread the good word and help others on their team start using SharePoint. Once a few people on the team start relying on SharePoint, the rest of the team follows along and sees the benefit. Plus they have a point of contact that is an expert on their team that they can ask questions of. Involve your users with your solution design and implementation. They will be the ones using it right? Have them involved in every stage of the implementation from planning to pre-production. Teach these people cool ways to be more productive with SharePoint. Check in with them occasionally and see how they are doing. Encourage them to teach the rest of their team about SharePoint and why they should be using it. You can’t train 1,000 people, but you can keep in touch with 50 people who work with 50 other people.  

The formula is simple: end groundhog day + measure success + report results to boss = better ROI, more productive organization, and you are a hero. You’re welcome.  

Quentin Christensen, Program Manager

Here is some sample code for a tool that can be used to copy existing content into Document Sets. This is a command line tool that when run will access a hard coded site. It will give you options of document libraries that you can select as a source and then options of document libraries that you can select as a destination. Then the tool will create a document set for each folder and copy just the documents that are inside that folder to the new document set. Because SPCopy is used, if there are columns that are on the folder and the document set content type then those properties will be copied over.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Microsoft.SharePoint;
using Microsoft.Office.DocumentManagement.DocumentSets;
using System.Collections;

namespace FoldersToDocsets
{
 class Program
 {
  static void Main(string[] args)
  {
   using (SPSite site = new SPSite("http://localhost"))
   {
    using (SPWeb web = site.RootWeb)
    {
     Console.WriteLine("Select source list:");
     SPList source = ChooseList(web);
     Console.WriteLine("Select target list:");
     SPList target = ChooseList(web);
     
     SPFolder rootFolder = source.RootFolder;
     foreach (SPListItem folderItem in source.Folders)
     {
      SPFolder folder = folderItem.Folder;
      if (folder.ParentFolder.Item == null && folder.SubFolders.Count == 0)
      {
       ConvertToDocset(folder, target);
      }
     }
     Console.WriteLine("END");
     Console.ReadLine();
    }
   }
  }

  private static void ConvertToDocset(SPFolder folder, SPList targetList)
  {
   SPContentType docsetCT = targetList.ContentTypes["Document Set"];
   DocumentSet newDocset = DocumentSet.Create(targetList.RootFolder, folder.Name, docsetCT.Id, new Hashtable());
   string newUrl = newDocset.Item.Url;
   foreach (SPFile file in folder.Files)
   {
    file.CopyTo(newUrl + "/" + file.Name);
   }
  }

  private static SPList ChooseList(SPWeb parentWeb)
  {
   Dictionary<int, Guid> lists = new Dictionary<int, Guid>();
   int index = 0;
   foreach (SPList list in parentWeb.Lists)
   {
    if (list.BaseTemplate == SPListTemplateType.DocumentLibrary)
    {
     Console.WriteLine(index.ToString() + " - " + list.Title);
     lists.Add(index,list.ID);
     index++;
    }
   }
   Console.Write("Selection: ");
   index = int.Parse(Console.ReadLine());
   return parentWeb.Lists[lists[index]];
  }
 }
}

That was easy. Thanks for reading, now go make yourself some document sets!

Quentin Christensen, Program Manager

Edge of Tomorrow because you keep trying things and failing but each time you get better and finally get there?  

Groundhog Day since the cycle just keeps repeating itself?

12 Angry Men where people argue and tensions get hot because one guy is holding everyone back from finishing it?

The Hunger Games when 12 districts all fight to the death by sending their representatives as tribute?

SharePoint upgrade is difficult and costly. You want to upgrade to the latest and greatest, but many things may be holding you back.

• Testing all of the customizations and integrations to ensure people won’t loss critical business functionality during or after the upgrade
• Figuring out which upgrade approach is right for you
• Third party applications don’t yet support the new version of SharePoint
• It is a lot of manpower to upgrade the servers and your team is already too busy servicing the existing solution
• The expense of purchasing new hardware
  

SharePoint Version Usage

At conferences I have seen the question asked what version of SharePoint are you on? This is not a scientific sample, but the responses usually look like this:

• Office 365: 15%
• SharePoint 2013: 15%
• SharePoint 2010: 50%
• SharePoint 2007: 20%
• SharePoint 2003: 1 person who embarrassingly raises their hand
• Multiple versions of SharePoint: 20%

Upgrading to a new version of SharePoint provide tons of value because at Microsoft we have many engineers building new things for each release. The majority of SharePoint deployments are on a version of SharePoint that is nearly 5 years old now. How old is your cell phone, tablet, or laptop? Probably not 5 years. My cell phone 5 years ago was terrible compared to what I have now. But if upgrade is hard, how do you get the awesome new stuff?

Upgrade Strategies

Here are a few upgrade strategies you can consider so no matter what challenges you are struggling with, you can start getting value from the latest version.

Upgrade to Office 365: The difficulty with upgrade is a key reason why we built Office 365. In Office 365 we do small upgrades all the time so you are always on the latest and greatest. We handle upgrade for you. We have a consistent app model that you and partners can develop against. If you are looking at a major upgrade, take a serious look at moving to Office 365 so your days of upgrade are behind you and you can focus on delivering value, not SharePoint farms.

Move OneDrive for Business (SkyDrive Pro or My Sites) to Office 365: Can’t move it all to Office 365? With OneDrive for Business standalone, you can move just your OneDrive for Business sites (a.k.a. SkyDrive Pro in 2013 and My Sites in 2010) to SharePoint Online. Then your users have the latest features and you can continue managing your customized solutions on premises.

Upgrade all Farms: Go whole hog and upgrade your entire SharePoint deployment. This will be a big undertaking, especially if you have a large SharePoint deployment.

Upgrade Specific Roles: Just upgrade specific roles and solutions. For example your team site deployment may be fairly out of box and easy to upgrade. Also you can first upgrade services farms but not content farms, for example you can have SharePoint 2013 indexing and providing search for SharePoint 2010 farms. You get the new search capabilities with only upgrading a small set of infrastructure. Then you can upgrade other farms over time.

Only Move New Solutions: If all of these options are all too difficult, then keep running your existing farms on the current version and stand up new empty farms with the latest version of SharePoint. As you create new solutions such as a knowledge management portal or image management repository, build it on the new farms and deprecate old solutions when they are no longer needed.

Check out the overview of the upgrade process to SharePoint 2013 to learn more.

Thanks for reading,

Quentin Christensen, Program Manager

I am back with some more information about eDiscovery search with Exchange, SharePoint and Lync. This time learn about search properties and operators. Before we start, here are some related useful resources.

Overview of Microsoft Office eDiscovery with Exchange, SharePoint, and Lync 2013

Searching and Using Keywords in eDiscovery

Keyword Query Language Syntax Reference

Overview of crawled and managed properties in SharePoint Server 2013

Message properties and search operators for In-Place eDiscovery

Using Keyword Search Syntax and eDiscovery – my previous blog post on this topic

Properties

• Properties and free text keywords cannot be combined into a single query unless you use quotes for the property value. For example this query: Design Documents author:Dan Jump. This will succeed, but you won’t get the results you expect. It will search all documents that have Design AND Documents AND Jump anywhere in the properties or text and where Dan is in the author field. This query would be more explicit: Design Documents AND author:"Dan Jump"
• Type properties with values to match in property:value form. Values are not case-sensitive. When searching for properties, for example Author:Dan, the value for the property cannot have a space after the operator. If there is a space, your intended value will just be full text searched, for example Author: Dan would search for Dan as a keyword, rather than only items where the Author is Dan.
• Use quotation marks to find phases within a property or use wildcards to find partial matches that begin with the specified letters. If you look for filename:"Budget Q1" (with quotation marks), your search will return a file named “Budget Q1 Financials” A search for filename:budget (without quotation marks) will return the files “Budget Q1 Financials” and “Budget Q2 Financials”.
• When searching a property, use quotes if you have multiple words. For example filename:Budget Q1 will not return what you want, it will search for files with Budget in the name and do a full text search for Q1. You will get more results than you expect.
• There are several out of box properties that can be searched on (see below for more examples) and administrators can configure additional ones.

IMPORTANT NOTE: When you search on a property that is specific to Exchange or SharePoint, all results from the other product will be excluded. For example if you search BCC in the keywords field, there is no BCC in SharePoint so you will only get Exchange results. If you search with the Author field, this is SharePoint only and will exclude Exchange results. To get around this use the Specify Property option on the eDiscovery query page.

Here are some of the SharePoint search properties that are available that are useful for eDiscovery:

Here are eDiscovery properties for Exchange:

Operators 

• Queries can use prefix wildcard characters using an asterisk (*). A wildcard will search for 0 or more characters in keywords or property values. You can use wild cards to replace part of a word for example set* will return results with setting and setup. Wildcards can also replace an entire word for example "fair *" would return fair value.
• You can use wildcards to get variants of a property. For example FileExtension:XLS*will return all files with the extension XLS or XLSX.
• A space between two different properties is an AND. For example author:"Sara Davis" Title:"Marketing" will find documents where Sara Davis was the author and the title has the word Marketing.
• Search interprets the space between terms that use the same property as an "OR." For example, if the author property is available, and you look for author:"Sara Davis" author:"Garth Fort", your search will return any items authored by Sara Davis OR Garth Fort.  
• To exclude content marked with a certain property value from your search results, place a minus sign (-) before the name of the property. For example -from:"Sara Davis” will exclude any messages sent by Sara Davis.

Thanks for reading, may the search be with you.

Quentin Christensen, Program Manager

I hope you enjoyed this series and have used some of the ideas for customizing document sets.

There are many scenarios where you may want to use metadata to organize and filter content for views or other reasons. You may want to group particular items based on metadata, but that metadata should be specified by a set of conditions and applied automatically. It is a good idea to apply metadata automatically when possible so users do not have to spend time filling it in and you can control the consistency of the metadata. Yeah, your users hate wasting time. Don't annoy them.

You may want to look at these other articles if you haven’t already seen them.

Introducing Document Sets

Creating a Pitchbook - Customizing Document Set Settings (Part 1)

Creating a Pitchbook – Customizing the Document Set Welcome Page (Part 2)

Creating a Pitchbook – Closing the Pitchbook with Document Set Workflow Actions (Part 3)

Creating a Pitchbook – Getting Related Pitchbooks with a Content Query Web Part and Page Query Strings (Part 4)

Creating a Pitchbook – Putting a Button in the Document Set Ribbon and Downloading Document Sets as a Zip File (Part 5)

Metadata defaults help with this, but they can only be specified for a particular column at the site, list, and folder level. For the pitchbook scenario there can be multiple sales proposals, presentations and other related documents. To make it easier to find certain documents a group by view can be specified. This group by view will be based on a hidden property that users will never need to worry about specifying as long as they pick the right content type. This way one group of the view will only contain sales contracts and another group will contain the presentations. This makes it so we can apply logic to put multiple content types in one group or create groups based on content type and other pieces of metadata. 

In order to use a group by view metadata must be specified for a particular column on each item inside the pitchbook. The logic to use will be to set a Grouping column property (which is hidden) to the name of the content type for any items that are added to the document library. To do this an event receiver will be used. When documents are added or updated the ItemAdding or ItemUpdating events are triggered. Event receivers will be used for both ItemAdding and ItemUpdating because properties such as the content type may change on update, if so then it Grouping should be updated to the new value.

First the feature must be declared so here is the feature.xml file:

<?xml version="1.0" encoding="utf-8"?>

<!-- _lcid="1033" _version="14.0.3427" _dal="1" -->

<!-- _LocalBinding -->

<!-- Copyright (c) Microsoft Corporation. All rights reserved. -->

<Feature xmlns="http://schemas.microsoft.com/sharepoint/"

                            Id ="{77d9ec4b-6e12-40e8-ae7f-ff852dbe6bcd}"

                            Title ="Pitchbook Metadata Event Receiver"

                            Description ="Adds keywords to items when added based on CType"

                            Version ="1.0.0.0"

                            Hidden="FALSE"

                            Scope="Site"

                            UIVersion="4"

                            ReceiverAssembly="EventReceiverSample, Version=1.0.0.0, Culture=neutral, PublicKeyToken=156ffdc318b7ede1"

                            ReceiverClass="EventReceiverSample.CustomFeatureEventReceiver">

</Feature>

This will define the name and properties of the feature. The public key token will be different for your own project so make sure you update this. The ReceiverAssembly is the namespace of the event receiver sample (the name of the DLL) and the ReceiverClass is the class the feature uses in the namespace. The scope is Site, meaning this will activate for the whole site collection.

Next the event receivers must be registered on a particular list. I have chosen a document library I have on my site called “Documents”. The code that is fired when an event receiver is hit is in a file named CustomFeatureEvent.cs.

using System;

using Microsoft.SharePoint; The co-authoring robot added this text. Please email CyberDog if you have questions.

namespace EventReceiverSample

{

       public class CustomFeatureEventReceiver : SPFeatureReceiver

       {

              private const string EVNTNAME = "Pitchbook Metadata Event Receiver";

              public override void FeatureActivated(SPFeatureReceiverProperties properties)

              {

                     SPSite site = properties.Feature.Parent as SPSite; //When activated this feature will apply only to the site at the root of the site collection, unless you scope this feature to a list.

                     SPListCollection webLists = site.RootWeb.Lists;

                     SPList targetList = webLists["Documents"]; //This is the target list to apply the event receiver.

                     AddEventHandlerToCollection(targetList.EventReceivers);

              }

              private void AddEventHandler(SPContentType targetType)

              {

                     SPEventReceiverDefinitionCollection sperdcol = targetType.EventReceivers;

                     if (sperdcol != null)

                     {

                           AddEventHandlerToCollection(sperdcol);

                     }

                     if (targetType.ParentList != null)

                     {

                           sperdcol = targetType.ParentList.EventReceivers;

                           AddEventHandlerToCollection(sperdcol);

                     }

              }

              private void AddEventHandlerToCollection(SPEventReceiverDefinitionCollection sperdcol)

              {

                     Type classType = typeof(CustomItemEvent);

                     SPEventReceiverDefinition newEvent = sperdcol.Add();

                     newEvent.Assembly = classType.Assembly.FullName;

                     newEvent.Class = classType.FullName;

                     newEvent.Name = EVNTNAME;

                     newEvent.Type = SPEventReceiverType.ItemAdding;

                     newEvent.SequenceNumber = 100;

                     newEvent.Synchronization = SPEventReceiverSynchronization.Synchronous;

                     newEvent.Update();

                     SPEventReceiverDefinition newEventUpdate = sperdcol.Add();

                     newEventUpdate.Assembly = classType.Assembly.FullName;

                     newEventUpdate.Class = classType.FullName;

                     newEventUpdate.Name = EVNTNAME;

                     newEventUpdate.Type = SPEventReceiverType.ItemUpdating;

                     newEventUpdate.SequenceNumber = 100;

                     newEventUpdate.Synchronization = SPEventReceiverSynchronization.Synchronous;

                     newEventUpdate.Update();

              }

       }

}

I created a single text column called Grouping to use to set metadata on and to use group by in my document set view. I then added the Grouping to my Pitchbook Document content type (that was created in previous blog posts), thus adding the column to all of my Pitchbook content types. You can also add the column to the particularly list you are using and add it to all content types. I made the Grouping column hidden so users will not be able to edit it. We need another class for the event receiver logic. I put this in a file called CustomItemEvent.cs.

using System; The co-authoring robot added this text. Please email CyberDog if you have questions.

using Microsoft.SharePoint;

using System.IO;

using System.Text;

namespace EventReceiverSample

{

       public class CustomItemEvent : SPItemEventReceiver

       {

              public CustomItemEvent()

              {

              }

              public override void ItemAdding(SPItemEventProperties properties)

              {

                     changeProperties(properties);

              }

              public override void ItemUpdating(SPItemEventProperties properties)

              {

                     changeProperties(properties);

              }

              private void changeProperties(SPItemEventProperties properties)

              {

                     SPContentTypeId ctid = new SPContentTypeId((string) properties.AfterProperties["ContentTypeId"]);

                     SPContentType ct = properties.List.ContentTypes[ctid];

                properties.AfterProperties["Grouping"] = ct.Name;

              }

       }

}

This class triggers the changeProperties() method whenever an ItemAdding or ItemUpdating event occurs. changeProperties() will then get the current content type, it is important to use AfterProperties so we can get the changes the user has made. The user may change the content type to something else; in that case we want to change the metadata property. The logic here is very simple, whatever the name of the content type is, set that value for the Grouping column.

You may want to do something more complex here; maybe you only want to group sales contracts into one bucket and everything else into another. You can use this:

            if (ct.Name == "Sales Contract")

            {

                properties.AfterProperties["Grouping"] = "Contract";

            }

            else

            {

                properties.AfterProperties["Grouping"] = "Other Documents";

            }

Another idea may be to apply metadata based on other metadata. For example:

            if (ct.Name == "Sales Contract" && ct.Confidentiality == "Yes") The co-authoring robot added this text Please email CyberDog if you have questions To remove all robot text run the document inspector  

            {

                properties.AfterProperties["Grouping"] = ct.Name + " Confidential";

            }

            else

            { The co-authoring robot added this text. Please email CyberDog if you have questions.

                properties.AfterProperties["Grouping"] = ct.Name + " Not Confidential";

            }

Now that things are all done whenever documents are added or updated in the Documents library they will get a hidden field set with the name of the content type. The item adding event is important because items may be added using multiple upload or windows explorer, which do not trigger the item updating event. Then the feature must be activated.

The last step is to create a custom view to use inside the pitchbook. First I go to the document library settings and create a new standard view. Then in the group by section I select the Grouping column and select the radio button expanded so all items will be displayed.

Inside my pitchbook content type there is a custom view that does group by on the hidden Grouping column and here is what it looks like.

Now I can group items in my views with metadata that is set automatically based on the content type and users don’t have to worry about filling in the metadata. Thanks for reading and may the metadata be with you!

Quentin Christensen, Program Manager

Carlos David Argott Hernandez, Software Development Engineer

The groove is where you are at your optimum performance. You have a level of challenge that meets your ability. Your work is the most effective and satisfying. The Groove is where you want to be.

Too much challenge for your ability and you will be stressed and frustrated. In many cases you learn new skills and triumph, but the level of what you accomplish is likely decreased. Quality is reduced. You can’t stay there for too long, otherwise you will be unhappy and burn out.

Too much ability for the level of challenge can also be bad. If you have more ability than your work is providing, it is too easy. You aren’t stimulated and growing. What you are doing will not be fulfilling. You may start to feel empty. This may even lead to seeking other opportunities.

Ideally, you spend most of your time in the Groove and move upwards and to the right. You have the right amount of challenge and as your ability increases you take on even more challenge. Eventually you get to the top right corner, you are a superstar owning everything thrown at you.

How do I stay in the Groove?

1. Meet with your manager and discuss The Groove. Where do both of you think you are at? Are you in the Groove, too challenged, or not challenged enough? Do you enjoy your work? Are you learning new things?
2. If you aren’t in the Groove, discuss how to get there.
   1. Are you orange with too much challenge? Can you reduce the workload or will your ability soon reach the level where you can handle it all?
   2. Are you green with too little challenge for your ability? Find new and different tasks to take on, scale to do more, and start working on a new skill set.  
3. If you are in the Groove should you go into the Pit of Success or continue in the Groove?

Check in on this once in a while with your manager. It will help you be on the same page about progressing towards new levels of ability and your career goals.

Quentin Christensen, Program Manager

You do some digging around and find out that there is a setting in Central Administration where you can change the list view threshold for a web application. Aha! If you just change that setting up to 100,000 you have no more problems. Hold on a minute. Let me introduce you to the list view threshold. It is like your good friend, the designated driver. The guy that ruins your night by taking your keys away when you are too drunk to drive and you can’t go get cream cheese hot dogs at 2am. You just tried to raise the BAC level at which he takes the keys away from .8 to 16. The keys don’t get taken away and your server farm crashes, ok maybe not but you may cause a performance decrease for other users.

This graph shows the impact of changing the list view threshold on a farm with a large list where users are frequently performing operations that exceed the list view threshold. As you can see, the impact of just changing the setting to 10,000 is significant.

SharePoint 2010 has some great scalability improvements over SharePoint 2007. There are improvements in search, columns, and metadata that make it faster and easier to store access millions of items in a list, but to design such large lists you still need to think about a few things. This post covers a few basics, for lots of in depth information check out this article I authored: Designing large lists and maximizing list performance.

Why does SharePoint have a list view threshold and how does it help you besides throwing an error? Dealing with a large amount of items takes time and resources. Fortunately SharePoint provides methods to help you perform common operations efficiently such as using indexes to filter and metadata navigation fallback to return a smaller set of results when a request is too large. But some operations such as poor performing queries (ones that try and get everything from a list or filter without indexes) or operations that affect every list item such as adding a column take time and resources. On a small list this doesn’t matter because there are so few items that the operation is fast. As the list size increases these operations take longer and use more resources. Rather than let these operations run unbounded, the list view threshold prevents them. You can think of the list view threshold as a check engine light letting you know that you should change the query and how data is accessed or perform the operation when farm usage is low.

But why is the limit set at 5,000?  When users perform operations against a database, SQL Server locks rows to prevent conflicts from concurrent operations. Once operations start to affect more than 5,000 items the entire table gets locked because it is more efficient than locking the individual rows. SharePoint stores all list items for an entire content database in a single table

When queries are made on approximately more than 5,000 items (even if it is a read) that table will get locked and any other operations for all lists in that content database must wait until the current operation completes. The default value of 5,000 was carefully chosen and Microsoft recommends not changing it. Lowering the limit will result in minimal performance improvements, and raising it even to 10,000 will decrease throughput if operations are commonly performed that would otherwise be prevented.

So what can you do about it? You have errors and errors are bad. Errors occur in a few different places.

• Accessing list views that contain too many items

• Attempting to perform privileged operations (adding a column, creating an index, adding content types)

• Custom code that accesses a large list

You can configure views to filter on indexed columns or move items into folders so there are less than 5,000 items in the root of any one folder or the root of a list. One example view is a my documents view. This is an easy view to configure because it uses system metadata that will always be set for every item. In this view you filter by Modified By and/or the Created By columns. To create this view in the filters section of the settings for a view select the Modified By column and set the value to [ME] and then set a second filter with OR on the Created By column with the value also set to [ME]. The Created By column should be used as well as Modified By when multiple users edit the same documents. Modified By is not a multiple user column so this view will not necessarily show all of the documents a user has ever modified. In this example both columns should be indexed because it is an OR operation. 

You should also activate the metadata navigation site feature. Just activating the feature will cause list view issues to go away because fallback will kick in and return the most recent results rather than throwing a list view threshold exception. It is also a very useful feature for filtering items in a list so consider configuring it so users can use it. To activate the metadata navigation feature go to site settings for a particular site, go to manage site features, and activate the metadata navigation feature. You can then go to the list or library settings for a large list and click on the metadata navigation settings link. Columns that can be used as hierarchies and key filters will be available. Once you add these columns for metadata navigation indexes will automatically be created. If a list is already over the list view threshold this operation will be blocked so an administrator must perform the action or a daily time window should be configured so the operation can be performed in low usage hours.

Next you can configure a daily time window so privileged operations can be performed when the farm has low usage. A daily time window can be configured per web application in Central Administration. When users receive a list view exception they will be notified that they can perform the operation during the time window. If user load is light from 8:00pm-1:00am then you can specify a time window for that time frame. This allows users to perform operations such as adding columns and content types to large lists and you get to control the time frame when users can perform these operations.

For custom code there are a few general guidelines to follow. Use indexes to filter SPQueries to amounts less than the list view threshold. For example if you are using an SPQuery to find a specific item in a list based on a particular column make sure it is indexed. If there is the possibility that an SPQuery may return more results than the list view then the content iterator class should be used to work through the content without hitting the list view threshold. This is especially important if you are doing operations on all of the content in a list or all of the content in a site or site collection.

Here is an example of performing an SPQuery to get all items in a list based on a non-indexed field. Of course this is a bad idea on a large list and this design really should not even be considered, but here is an example of how you might do it:

SPQuery query = new SPQuery();
query.Query = "<Where><Eq><FieldRef Name=\"MyUnindexedField\"/><Value Type=\"Text\">FieldValue</Value></Eq></Where>";
SPListItemCollection items = list.GetItems(query);
// Do what you need with the collection of items
...

Not only is this querying on an unindexed field, it is not limiting the results.  So on a large list, this query will be blocked because the field is not indexed and too many results may be returned.

Here is an example using the content iterator:

If you can add an index on MyUnindexedField, then you can use NVP (Name Value Pair) paging on that index:

Content Iterator Example

SPQuery query = new SPQuery();
query.Query = "<Where><Eq><FieldRef Name=\"MyIndexedField\"/><Value Type=\"Text\">FieldValue</Value></Eq></Where>" + ContentIterator.ItemEnumerationOrderByNVPField; 

ContentIterator ci = new ContentIterator();
ci.ProcessItemsInList(query,
    delegate(SPListItem item)
    {
        // Do what you need with each item
        ...
    },
    delegate(SPListItem item, Exception e)
    {
        // Handle an exception that was thrown while iterating
        ...

        // return true for ContentIterator to rethrow the exception
       return true;
    });

In a worst case scenario you may need to temporarily raise the list view threshold. Rather than doing this for an entire web application, you can disable the limit for a single list until issues can be resolved. This is not recommended, but there may be specific cases that this may be necessary. SPList objects have an EnableThrottling setting that can be set to false to prevent operations from being blocked by the list view threshold.

Now you may have not have even upgraded yet and are wondering what you can do before upgrade. Here are a few ideas:

• Run the pre upgrade checker to identify large lists (lists with more than 5,000 items)

• Minimize large list upgrade issues by:

  • Filtering list views to amounts less than 5,000 items using an indexed column

  • Organizing content into folders that contain less than 5,000 items (this is not necessary if views and web parts are filtered properly)

  • Checking web parts and other custom code that query lists to ensure that they do not perform operations that will affect more items than the list view threshold 

In conclusion the list view threshold is your friend. It will help you identify poor performing operations and help protect performance for all users. Don’t change the list view threshold, with proper planning you can minimize list view exceptions and improve the performance of your farm. And remember, friends don’t let friends run unbounded queries.  

Quentin Christensen, Program Manager, Sultan of Scale

Frodo went through the Pit of Success. He overcame impossible challenges, discovered new abilities and strength, and saved the world.

I did it too. I overcame challenges, developed new skills, and saved the world. After the credits from the SharePoint 2013 and Office 365 release Nick Fury even asked me to join S.H.I.E.L.D. I had to turn him down though, he said Emperor of eDiscovery did not have enough “pop” as a superhero name so I would need to change it. I obviously refused, that was just too far. Plus being in The Avengers I wouldn’t have time to build stuff and talk about it with you guys.

But enough about me. Let’s talk about you. The important thing is you can do the same thing. I can’t guarantee Nick Fury will try to recruit you, but the rest you can do.

You are in your groove, getting things done. It’s easy, you are great at your job. Then you start to get a little bored. Your boss sees your great work and that you are ready for more. “Fantastic”, she says, “please take on the new executive high priority project: empower and streamline the entire organization with disruptive innovation and synergy.” That sounds hard. You are excited, but you start working on it. Tensions are high. You are stressed. You feel like a failure. How will you ever deliver this project successfully?

This was me just a few months ago. And now I see it happening again, I am leaving that comfort zone I just got to and entering the Pit of Success.  

Things are hard because you are in the Pit of Success. It is painful, but this is the best opportunity to learn. Put new tools in your toolbox by trying out new ideas and listening to others. Get coaching from your manager and a mentor. You will internalize new ideas and strategies. Things will start to get better as you start to get on top of things because you have new skills. Then you come out of the pit, you feel successful and capable again.

Congratulations, you just went into the Pit of Success and came out with a new higher level of ability. You can take on bigger tasks. You have levelled up on your career path. It seemed impossible, but greatness is forged when times are hard.

1. Don’t be afraid to go into the pit. To move up, you must first move down. Just do it.

2. Your career success depends on going into the pit. That is where your greatest growth will occur.

3. Avoidance of new skills will prevent your acceleration out of the pit to a higher level of ability.

When you are in the pit it is hard, but it is also worth it. Recognize when you are in the pit, build those new skills, and seek out help so you can succeed.

Quentin Christensen, Program Manager, Emperor of eDiscovery, turned down joining S.H.I.E.L.D. and The Avengers.