The Distributed Cache Service

SharePoint’s distributed caches are hosted and maintained by the Distributed Cache Service, itself a thin wrapper over a Windows Server AppFabric cluster. Understanding SharePoint’s Distributed Cache Service requires an understanding of AppFabric together with a few details on SharePoint’s implementation. Most of the details here apply equally to non-SharePoint AppFabric cluster as well; just view SharePoint as a particular implementation example.

AppFabric Physical Architecture

Key details of AppFabric’s physical architecture are described here and represented in this diagram (from the same location).

In SharePoint’s implementation, web and service applications are the “Cache-enabled application servers (cache clients).” The “Cache Servers” are SharePoint servers where the Distributed Cache Service Instance has been installed and enabled, and the “Cluster configuration storage location” is the CacheClusterConfig table in the SharePoint Configuration Database.

The CacheClusterConfig table in the Configuration Database stores configuration items as typed Key/Value pairs using a custom ICustomProvider implementation. The original values are XML snippets describing caches, cache hosts, and other properties. They are serialized and converted into byte arrays for storage, but most data can be deserialized and viewed using the Export-AFCacheClusterConfiguration cmdlet.

AppFabric Cache Hosts in SharePoint

Management of Distributed Cache Service Instances (AppFabric Cache Hosts) in SharePoint is different than management of most SharePoint service instances. Most service instances always remain installed on servers in the farm, whether online or not. These service instances are like Windows services, which are always installed on a server whether they’ve been enabled or not. For example, the User Profile Sync Service Instance is typically only online and running on one server in the farm, but it’s installed – and disabled - on all servers. To see a list of all service instances installed on a given server, both online and disabled, run the Get-SPServiceInstance cmdlet, using the –Server parameter to limit results to a particular server. The Services on Server page in Central Administration displays the same information.

Unlike other service instances, though, the Distributed Cache Service Instance should either be installed *and* online on a SharePoint server, or not installed at all. If the service instance is stopped (disabled) but not uninstalled, details about the associated Cache Host stay in the Cache Cluster Config store, which can cause problems.

For this reason, the Distributed Cache Service Instance should never be stopped via the Services on Server page in Central Administration or via Stop-SPServiceInstance in PowerShell. A special cmdlet, Remove-SPDistributedCacheServiceInstance, is available to stop *and* uninstall the local Distributed Cache Service Instance from a SharePoint server. This cmdlet, and its complement Add-SPDistributedCacheServiceInstance, should be used instead of Stop- and Start-SPServiceInstance for managing the local Distributed Cache Service Instance.

By default, the Distributed Cache Service Instance is installed on every SharePoint server when it’s joined to a farm. If you prefer to not install the Distributed Cache Service Instance at join time, you can specify -SkipRegisterAsDistributedCacheHost when running the Connect-SPConfigurationDatabase or New-SPConfigurationDatabase cmdlets. Note that at least one server must be running the Distributed Cache Service Instance for the farm to function properly.

For all AppFabric clusters, a simple command for listing all known cache hosts in the cluster is Get-AFCacheHostStatus. Don’t forget to run Connect-AFCacheClusterConfiguration before running other AppFabric cmdlets. To retrieve a list of SharePoint servers running the Distributed Cache Service Instance, you can run the following PowerShell command:

PS:> Get-SPServer | ? {($_.ServiceInstances | % TypeName) -contains 'Distributed Cache'} | % Address

We’ll discuss more details about cache hosts soon, but first let’s discuss AppFabric’s logical infrastructure.

AppFabric Logical Infrastructure

Details regarding the logical infrastructure of an AppFabric cluster are provided here and in this diagram (from the same location):

The basic logical entity in an AppFabric Cache Cluster is a Named Cache (frequently just called a Cache). A Named Cache is a container for cached objects. The ten SharePoint caches listed before are each AppFabric Named Caches. As illustrated here, Named Caches span all hosts in the cluster, distributing items for storage across allocated memory on all servers; however, cached items within a Named Cache are stored only once (by default). This is an important consideration when planning cache infrastructure, so let’s spell it out again: by default (and in SharePoint), cached items in an AppFabric Named Cache are stored only once across the entire cluster. If the cache host storing that cached item crashes or is shutdown non-gracefully, that item is no longer available in the cache.

In the previous paragraph we introduced cache items. Items are often collected and stored in a Region, which is a sub-collection of cached items within a specific Named Cache. Storing items in a shared region can make retrieval of the entire related collection easier. Like individual cached items, though, regions exist on only a single host in the cluster by default (and in SharePoint). So if the server hosting the region is lost, all items in the region are lost with it. Also, note that all items in the Region, and the Region itself, are stored on a single cache host.

Both individual cached items and regions can co-exist in the same named cache. I believe that in SharePoint all cached items are contained within regions.

To list all named caches and regions in a cluster, run Get-AFCache | Format-Table –AutoSize. For a list of all caches, run Get-AFCache | Format-Table CacheName.

With the infrastructure described, let’s dive more deeply into configuration of caches and cache hosts.

Cache Configuration Details

As we consider configuration details for caches and cache hosts, our conversation will be dominated by memory management, resiliency, and availability issues. We’ll discuss expiration and eviction of items, throttling of requests, and redundant storage. As before, we’ll discuss concepts in general and use SharePoint as a specific example.

Later, we’ll discuss configuration of cache hosts, but first, let’s begin by discussing configuration of individual caches.

To view common configuration details for individual caches, run the following command:

PS:> Get-AFCache | % {Get-AFCacheConfiguration -CacheName $_.CacheName}

Output for the ViewState cache is displayed here:

CacheName                : DistributedViewStateCache_f3bd4763-f482-4bb8-a5a5-f40806460bdd
TimeToLive               : 10 mins
CacheType                : Partitioned
Secondaries              : 0
MinSecondaries           : 0
IsExpirable              : True
EvictionType             : LRU
NotificationsEnabled     : False
WriteBehindEnabled       : False
WriteBehindInterval      : 300
WriteBehindRetryInterval : 60
WriteBehindRetryCount    : -1
ReadThroughEnabled       : False
ProviderType             :
ProviderSettings         : {}

The meanings of each of these properties is as follows.

• CacheName: The internal name of the cache.
• TimeToLive: The default time span until expiry for cached items. Note that this can be overridden for any individual item, and in SharePoint there are different standard TTLs used for items in each named cache, as in the below table. Time until expiry has an important impact on eviction and memory management which will be discussed in the section on cache host configuration.

• CacheType: How data is stored in the cache’s storage medium. Partitioned is the only option.
• Secondaries: How many additional replicas of cached data are to be stored. Additional replicas provide redundancy and resiliency for cached items, as replicas are always stored on a different storage node. This is always 0 in SharePoint, where high availability is not currently supported.
• MinSecondaries: The minimum number of secondaries which must be online to allow writing to the cache. By default, it’s the same as the number of secondaries configured for the cache. Always 0 in SharePoint, where there are no secondaries.
• IsExpirable: Whether items in the cache are to be evicted after their TTL passes. Always True for SharePoint caches.
• EvictionType: The algorithm used to evict non-expired items when a cache’s high watermark is passed. Can be set to LRU (Least Recently Used) or None. For most caches in SharePoint, this is set to LRU. For the ActivityFeedLMT cache, this is set to None. See the section on cache host configuration for details on eviction.
• NotificationsEnabled: Whether the cache will notify subscribers when cached items are changed or deleted. Always False in SharePoint caches.
• Read-Through and Write-Behind properties: The remaining properties specify details on Read-Through and Write-Behind for the cache. For details on RTWB concepts, see this MSDN article. SharePoint caches don’t utilize RTWB.

Most SharePoint caches have the same configuration. However, run this command to note that the ActivityFeedLMT cache has an EvictionType of None.

PS:> Get-AFCache | % {Get-AFCacheConfiguration -CacheName $_.CacheName} | Format-Table CacheName, EvictionType

Note that there are no quota-related properties specified at the cache level for SharePoint caches, or by default for any caches.

You may want to know how many items are stored and how much memory is in use for individual caches. For specific stats about each cache, run this command:

PS:> Get-AFCache | % {
    $CacheName = $_.CacheName
    Get-AFCacheStatistics -CacheName $CacheName | Add-Member -MemberType NoteProperty -Name 'CacheName' -Value $CacheName -PassThru
}

I’ve added a little formatting and cleanup to get detailed information about each cache together with its name. You could pipe the output from this command to Export-Csv to create a short report. Typical output is shown here:

CacheName         : DistributedLogonTokenCache_f3bd4763-f482-4bb8-a5a5-f40806460bdd
Size              : 36864
ItemCount         : 6
RegionCount       : 6
RequestCount      : 55
ReadRequestCount  : 26
WriteRequestCount : 14
MissCount         : 32
IncomingBandwidth : 96924
OutgoingBandwidth : 880

Having illustrated key elements on configuring individual caches, let’s move on to configuring cache hosts.

Cache Host Configuration Details

To retrieve configuration details for cache hosts, run the following command, which retrieves information about each host currently in the cluster:

PS:> Get-AFCacheHostStatus | % {
    $Status = $_.Status
    Get-AFCacheHostConfiguration -ComputerName $_.HostName -CachePort $_.PortNo |
        Add-Member -MemberType NoteProperty -Name 'Status' -Value $Status -PassThru
} | Format-List -Property *

I start this command with Get-AFCacheHostStatus since it returns all hosts in the cluster without further parameters necessary, unlike Get-AFCacheHostConfiguration. Note however that Get-AFCacheHostStatus attempts to ping each host in the cluster in order to report on status, and the timeout for this ping is 10 seconds. For a faster version of this command, at least for SharePoint servers, try this:

PS:> $SPDCServers = Get-SPServer | ? {($_.ServiceInstances | % TypeName) -contains 'Distributed Cache'} | % Address
PS:> $SPDCServers | % {Get-AFCacheHostConfiguration -ComputerName $_ -CachePort 22233}

Of course, you won’t get a Status without Get-AFCacheHostStatus.

Output from the first command will look like the following:

Status          : Up
HostName        : SERVER09.gavant.local
ClusterPort     : 22234
CachePort       : 22233
ArbitrationPort : 22235
ReplicationPort : 22236
Size            : 600
ServiceName     : AppFabricCachingService
HighWatermark   : 99
LowWatermark    : 90
IsLeadHost      : True

Let’s describe the meaning of each of these properties:

• Status: If the host responds to a standard ICMP ping within 10 seconds, this reports the status of the AppFabric service on that server. May be: {Up, Down, Starting, Stopping, ShuttingDown, Unknown}. This is the output from Get-AFCacheHostStatus.
• HostName and ServiceName: Host and service name.
• CachePort: The main port for public (external) communication with the cache host and cluster. Must be open to incoming client traffic.
• ClusterPort, ArbitrationPort, and ReplicationPort: Used for internal data management communication amongst the hosts in the cluster. Must be open between servers.
• Size: The amount of memory in MB to be allocated for live cached items. Note that actual memory used by the process will be significantly greater than this amount, and will be discussed later.
• LowWatermark: Percentage of memory usage (from Size) when *expired* items are removed (evicted) from cache if expiration is enabled.
• HighWatermark: Percentage of memory usage (from Size) when *all* items may be removed (evicted) from cache if eviction is enabled.
• IsLeadHost: Whether this host is a lead host for cluster management. Not used in SharePoint.

Now that we’ve briefly discussed each of these properties, let’s dive deeper into their implications.

Eviction, Expiration, and Watermarks

The time has come to explain eviction, expiration, and watermarks. In a nutshell, the goal of the AppFabric service is to maintain the memory a cache host uses to store cached items between the low watermark and high watermark configured for that cache host. Every one second (by default), current memory usage for the host is checked against the Size and computed high and low watermark values for the host; based on the results, the following memory management algorithm is implemented.

• Low Watermark not yet reached. No items are removed from the cache, even if expired.
• Low Watermark reached, High Watermark not reached. Expired items are evicted, but non-expired items are not.
• High Watermark reached. Expired and non-expired items are evicted until low watermark is reached.

This is shown graphically here:

If and when less than 15% (by default) of server memory remains, an eviction run is initiated regardless of the local cache host’s watermark and size settings. That is, even though the host is not using all of its allowed memory, if available memory on the server is below 15% of the total physical memory, a full eviction run will begin, as if the high watermark had been passed.

Note that caches specify whether they will be subject to expiration and eviction via the IsExpirable and EvictionType properties, as discussed in the previous section. If either of these excludes the cache from eviction, corresponding cached items will not be removed. For example, if IsExpirable is True and EvictionType is None (as it is for the Activity Feed LMT cache in SharePoint), expired items will be removed once the cache reaches its low watermark, but non-expired items will never be removed. As a result, if no items in the cache are expired, nothing will be removed.

If throttling is enabled (see below) the cache would eventually be write-throttled and no further items would be added till some items expired or were removed. However, throttling is not enabled by default in AppFabric for Windows Server and *I believe* that in this case the cache will continue to grow beyond its allotted size, governed only by the algorithm described above. Test and consider this when planning your own cache and host configurations.

Cache Statistics

Since the current amount of memory in use by the cache is so important, you’ll be interested in the commands which return a snapshot of current usage. As before, you have a couple options for retrieving information about all servers in the cluster, one using the Get-AFCacheHostStatus to iterate through all hosts, and one by finding all configured service instances in SharePoint. They are as follows:

PS:> Get-AFCacheHostStatus | % {
    $ServerName = $_.HostName
    Get-AFCacheStatistics -ComputerName $_.HostName -CachePort $_.PortNo | Add-Member -MemberType NoteProperty -Name 'ServerName' -Value $ServerName -PassThru
} | Format-List -Property * 

PS:> $SPDCServers = Get-SPServer | ? {($_.ServiceInstances | % TypeName) -contains 'Distributed Cache'} | % Address
PS:> $SPDCServers | % {
    $ServerName = $_
    Get-AFCacheStatistics -ComputerName $_ -CachePort 22233 | Add-Member -MemberType NoteProperty -Name 'ServerName' -Value $ServerName -PassThru
}

And typical output looks like this:

ServerName      : SERVER09
Size            : 51200
ItemCount       : 18
RegionCount     : 13
NamedCacheCount : 10
RequestCount    : 393
MissCount       : 66

Here, the SERVER09 cache host has allocated 51200 bytes (about 51K) to cached items. Since the allowed size for this cache host is much higher than 51K, neither expiration or eviction is necessary on this server, assuming at least 15% of the server’s physical memory is free.

Throttling

According the this MSDN article, AppFabric for Windows Server is subject to throttling based on percentage of server memory in use and percentage of AppFabric service memory in use. However, in my investigations I’ve found that throttling is disabled by default in AppFabric for Windows Server, and SharePoint does not change the defaults. This means that even though no memory from the cache host’s allowed size remains, or even if no memory on the entire server remains, the system will continue to try to serve requests and allocate memory for cached items.

I’ll continue to investigate Throttling and update this section if I find other information. Please let me know if your testing reveals different behavior than I’ve described.

With all this discussion of memory algorithms and calculations, you won’t be surprised to learn that memory overcommitment schemes employed by virtualization hypervisors, such as Hyper-V’s dynamic memory, are not supported with SharePoint, and not recommended with AppFabric in general.

SharePoint Cache Service Details

Now that we’re explained most aspects of AppFabric cache and cache host configuration, let’s explore some of the defaults used for SharePoint cache hosts, and some recommendations.

Cache Host Size for SharePoint Hosts

There are two points during setup of a Distributed Cache Service Instance on a SharePoint server when the local cache host configuration comes into play: at service installation (e.g. during Farm Join or when running Add-SPDistributedCacheServiceInstance) and at service provisioning (e.g. immediately following service installation, or when calling Start-SPServiceInstance).

At service installation time, the Size property for the local cache host is set to 5% of the total physical memory of the host. For example, if 16GB of physical RAM are installed on the host at installation time, the size of the local cache host will be set to 800MB. Note that this value could be different on each SharePoint server if they have different amounts of physical RAM at installation time. Note also that this value won’t automatically change if the amount of physical RAM allocated to the server changes.

At service provisioning time, SharePoint checks that the amount of available physical memory in the server is at least 100MB more than the allowed size for the cache host. So if, as in the above example, the cache host size is set to 800MB, at least 900MB of physical RAM must be available at provisioning time, or the service will fail to start.

These default values may not be appropriate for your environment. In the next section we’ll discuss options for changing them.

Changing Cache Host Size for SharePoint

There are two options for modifying cache host size for SharePoint AppFabric servers. Both require shutting down the entire cluster (all cache hosts).

The first utilizes AppFabric’s own PowerShell cmdlets. First, run Stop-AFCacheCluster to shut down all hosts in the cluster, then on each cache host run Set-AFCacheHostConfiguration -CacheSize <NewSizeInMB> to specify a new cache size. Don’t forget to run Start-AFCacheCluster to restart all hosts in the cluster. You can also specify different high and low watermarks with the Set-AFCacheHostConfiguration cmdlet.

Advantages of the native AppFabric cmdlet approach are that 1) all hosts in the cluster are stopped immediately (not gracefully), 2) you can specify different cache sizes for each host, and 3) you can configure low and high watermarks if necessary.

The second approach is to use SharePoint’s Update-SPDistributedCacheSize cmdlet. This takes only one parameter, -SizeInMB. It shuts down all hosts in the cache cluster, updates the cache size for each of them, then restarts them all.

Disadvantages of the SharePoint cmdlet are that it 1) shuts down all but the last host in the cluster gracefully and 2) sets all cache hosts to the same size. Graceful shutdown takes much longer than immediate shutdown, since all cached items must first be moved to a different host. Yet since the entire cluster is to be shut down, graceful shutdown is not helpful here.

Hopefully, Microsoft will address the graceful shutdown issue in the future, and setting all servers to the same memory size may be appropriate in your farm. If these items aren’t concerns for you, I’d recommend using the SharePoint-specific cmdlet, as this will always command more respect if and when you need support.

Whether you use AppFabric or SharePoint cmdlets to modify cache host size, note that if you uninstall and reinstall the Distributed Cache Service Instance on a server (i.e. by running Remove-SPDistributedCacheServiceInstance and then Add-SPDistributedCacheServiceInstance) the cache host size will be reset to the default (5% of physical memory at time of installation). If removing and adding the cache service instance is part of your maintenance cycles, make sure to also modify the cache size afterwards if needed.

We’ve discussed many concepts relevant to AppFabric infrastructure and planning. Now let’s focus on a couple specific applied details.

Planning Server Memory and Cache Host Size

A key detail in AppFabric Cache Service planning is that the actual memory usage of the DistributedCacheService process will be significantly larger than the size allocated in the cache host configuration. This MSDN article states that at least twice the amount of memory specified for the cache host will be used by the process due to memory management algorithms. Run this command for a quick review of how much memory the process is actually consuming:

PS:> Get-Process DistributedCacheService | fl *64*

At the time I ran this command, Get-AFCacheStatistics reported my cache host size as 9216 bytes. The process’s Working Set, however, was 584118272 bytes (almost 600MB). This is of course much more than twice the used amount of RAM; my assumption is that there is always a base level of memory overhead no matter how small the actual caches may be.

The key takeaway here is that if you have a certain amount of memory to allocate for AppFabric, the size specified for the host configuration should be half of that. For example, if I intend to allocate 16GB of physical memory for AppFabric, the size specified in the host configuration should be 8GB.

Also in that article, it’s recommended not to allocate more than 16GB for the AppFabric server (and corresponding 8GB for the cache host configuration). If the cache host’s size is larger than 16GB/8GB, garbage collection could take long enough to cause a noticeable interruption for clients.

A common recommendation is to spec AppFabric servers with 16GB of physical RAM, and set the cache host size to 7GB. With this arrangement, you can expect about 14GB to be used by the AppFabric process, leaving 2GB for other server processes on the host.

Let’s wrap up by discussing a couple additional considerations specifically relevant to SharePoint’s AppFabric implementation.

Other Considerations for AppFabric in SharePoint

High Availability

This one is easy - SharePoint (as ofMarch 2013) does not provide any high availability for its caches. As briefly discussed above, this means that each item and region in SharePoint’s named caches exists only once across all the memory in the cluster. If the server where that item has been stored in memory is lost or shut down ungracefully, that cached item will be lost. As discussed at the very beginning of this post, this is generally not a problem for cached items because they are authoritatively stored elsewhere. Nevertheless, there are a couple things to keep in mind.

First, retrieving cached items all over again involves a performance hit, the very hit the caches are intended to help avoid. There could be interruptions and delays while the caches are being refilled. For example, if the ActivityFeed cache is lost, users may not see all recent updates in their Newsfeed, or may see the “We’re still gathering the news” message as the cache is repopulated.

For the ActivityFeed and ActivityFeedLMT cache, there are two PowerShell cmdlets to manually begin repopulation of the caches before users actually request data. These are Update-SPRepopulateMicroblogLMTCache and Update-SPRepopulateMicroblogFeedCache. In situations where maintenance leads to loss of these caches, plan to run these cmdlets immediately afterwards to repopulate data manually.

A second concern when cached data in SharePoint is lost is that some items in SharePoint are *only* stored in the cache; specifically, updates regarding followed documents are only stored in the cache (as of March 2013). If these cached items are lost they won’t be able to be regenerated and will no longer appear in users’ feeds.

To avoid losing items from the cache and/or having to retrieve them again, you can use the Stop-SPDistributedCacheServiceInstance cmdlet with the -Graceful switch. This will move all cached items from the local cache host to other cache hosts in the cluster. For this to be effective, there must be space on the other servers to accommodate these items. Also note that if shutting down the entire cluster, such as to change the cache host size, there’s no way to avoid losing all of the caches and items. Plan accordingly.

Other Caches in SharePoint’s Deployment

One last detail is that Microsoft has stated that additional named caches should not be deployed to the SharePoint AppFabric cluster (i.e. by using the New-AFCache cmdlet). If you need a cache for a custom solution, you’ll need to deploy a separate AppFabric cluster (or server) and create the cache there. Then point your solution at the external AppFabric cluster. There’s also no supported way to add your own cached items to SharePoint’s named caches.

Conclusion

This concludes our presentation on AppFabric and SharePoint’s Distributed Cache Service. I hope it provides both SharePoint and AppFabric administrators with a deeper understanding and greater ability to manage distributed cache clusters.

To assist with scaling and performance, SharePoint 2013 utilizes Windows Server AppFabric to maintain an in-memory cache of frequently-used data. While this in-memory cache achieves its goals and does improve scalability, it also adds another service to maintain to SharePoint’s long list of supporting elements. In this article I’ll summarize and distill information on AppFabric from across MSDN and TechNet and add a SharePoint tinge to it to help you prepare for and manage SharePoint’s Distributed Cache service.

If you’re an AppFabric admin, there’s a lot of detail here that will help you as well. Just view SharePoint’s AppFabric cluster and caches as an example of an AppFabric implementation.

We’ll begin with the fundamentals to ensure we all share the same basic understandings.

What is a cache?

As I said, we'll start with the fundamentals :). A cache is a local, non-authoritative copy of data created and/or stored authoritatively elsewhere. Because retrieval of the data from its authoritative source is an expensive operation, instead of retrieving it freshly each time it’s needed, previous copies of the data are retained and used for subsequent needs.

Reasons to avoid the authoritative operation include:

• Network latency and delays. The data may be stored across relatively slow network links or require multiple intermediate hops to be retrieved. Rather than retrieving it for each request, a copy is stored nearby. An example of this in SharePoint is the BLOB cache; BLOBs from content databases are stored locally on front-end servers to avoid the need for further network trips to SQL.
• User mediation required. The user may have to participate in retrieving the data, such as via authentication prompts and browser redirects. Imagine if a user had to undergo the three-legged process of WS-Federation authentication for every request! Far simpler to cache a security token and rely on it for subsequent requests.
• Processing power required. The authoritative data may be generated locally, but a relatively high amount of processing power is needed to generate it. Instead of regenerating the data for each request, a copy of the processed data is retained for a while. A good example of this is ASP.NET’s page output cache, which avoids ASP.NET having to fully process each request. A SharePoint-specific example is the SiteData Object Cache, wherein the results of cross-site list item queries are cached and used for subsequent requests to avoid the processing hit of running the query again.
• Authoritative source not consistently available. Perhaps the client or server cannot be consistently connected. If the client depended on always communicating with the data’s authoritative source, it wouldn’t be able to work when not connected. Caching the data locally mitigates this. SkyDrive Pro is an example of offline caching.

It’s important to call out that in all cases of caching, the expense of generating or retrieving the authoritative data is traded for greater memory requirements where the data is cached. Either RAM (short term) or hard disk (long term) memory will need to be allocated for the data.

Implications of Caching

The defining characteristic of a cache is that it isn’t the authoritative source of the data. This leads to some important ramifications:

1. Cached data may not be completely current. Results and data may have changed at the authoritative source and not yet be reflected in the cached data. This effects many cache configuration choices, such as how long the cached data should be relied upon. There may be cases where the cache should be completely bypassed, or when a cache shouldn’t be used at all.
2. Caches generally don’t need to be as resilient as the authoritative source. If cached data is lost, it can be retrieved or regenerated, albeit at an expense. Again, this effects decisions about cache configuration, such as redundancy and high availability. Since the cache can be regenerated with a minor performance hit, secondary copies and resiliency may be less important. Then again, they may be quite important, depending on how difficult regenerating the data could be.

Having discussed shared characteristics of all caches, let’s discuss some of SharePoint’s caching mechanisms.

Caches in SharePoint

Caches abound in SharePoint – from disk-based caches of data from databases, to in-memory caches of processed queries and pages, to caches of properties retrieved from internal and external services. The Distributed Cache service in SharePoint and its AppFabric underpinnings are just one example of a SharePoint cache. To emphasize this point, let’s begin by discussing some caches that have been around in SharePoint for a little while already.

Some common SharePoint caches include the BLOB cache, the ASP.NET page output cache, and the User Profile Service properties cache. We’ll describe them one by one to help you understand caches and their implications better.

BLOB Cache

Or Binary Large Object Cache. My wife always perks up when I bring up the cash blobs in SharePoint, at least till I start explaining them. If you’re reading this, you’re more interested in knowing that when a user requests a byte-stream object in SharePoint, such as a document or an image, and the item is retrieved from a SQL database to be sent to the user, the BLOB cache stores this object on the local file system of web servers. This avoids the need to retrieve the data from SQL again for subsequent requests for the same object.

Using this familiar example, let’s call out some of the advantages and disadvantages of the Blob Cache.

1. The key advantage is avoiding the network latency of retrieving large files from across the network from SQL. In exchange for avoiding this trip, additional disk space is needed on the front-end servers. Note also that if the speed of retrieving data from disk isn’t faster than retrieving data from across the network, there’s limited benefit to this cache.
2. Data could be stale. A user sending a request to one front-end server might update the large file, but a second user working on a different server could still be served an older copy cached on that server. There are mechanisms in place to check for updated files and deprecate the cached version, but there will always be points in time where the cached data is stale.
3. The data is cached separately on each server. If a user requests the BLOB via one front-end server, the BLOB will be cached only there. If another user requests the same BLOB via another front-end server, that server will have to retrieve the BLOB itself, and store it again in its own file system. Note that AppFabric and the Distributed Cache are designed largely to mitigate this issue of cache locality (though not for the BLOB cache).
4. The cache is less resilient than the database. If you lose the cached blobs, no critical data loss will occur. As such, you only back up and provide high availability for the database, not the cache.

Now let’s move on to the ASP.NET Page Output Cache.

ASP.NET Page Output Cache

Another well-known cache in SharePoint is the ASP.NET Page Output cache, which is actually a standard ASP.NET feature. It provides a good example of an in-memory cache designed to save processing power. Every incoming HTTP request causes activity and requires CPU cycles for ASP.NET’s HTTP handlers. The Page Output cache allows the results of this activity to be stored and re-used, avoiding the need for repeated processing of identical or similar requests. While for a request or two the performance gains may be small, for sites processing hundreds or thousands of requests, avoiding repeated processing saves a lot of resources and vastly improves scalability.

Let’s call out advantages and disadvantages of the Page Output cache.

1. The key advantage is avoiding the use of processing power to process identical requests many times. In exchange for avoiding this CPU usage, memory is required to store the page output.
2. Data could be stale. If the page’s processing retrieves data from a web service or some other changing resource, that data might have changed since the page was last processed, but the changes won’t be reflected in the cached page. An example I recently came across is the time zone entries in Wikipedia (e.g. US Eastern Time Zone). There’s a “current time” row in the information box on the page, but the time is usually stale. You can click “Refresh the Clock,” which apparently forces Wikipedia’s servers to discard the cached version of the page and generate a new one.
3. Data could be inaccurate. This is a tricky one and one which a software architect would do everything to avoid. When data is processed for the sake of one request and then served to fulfill another, there’s a possibility that the context of the latter request isn’t exactly the same as the former. Generally, you’d account for significant changes of context by caching different versions of the data, such as one for authenticated users and one for anonymous users. There remains a possibility that differences in context could slip through, though.
4. More RAM is required. This of course depends on how many requests are cached. Each one requires its own space in RAM (or perhaps hard disk). CPU resources have been saved, but if memory isn’t accordingly increased the exchange might not be worthwhile.

Let’s go through one more example of a cache in SharePoint, one which probably is not so familiar.

User Profile Application Proxy Settings Caches

User Profile Service Application (UPA) Proxies retrieve configuration details from their associated UPAs and databases and cache those details in several per-process in-memory caches. Settings stored in these caches include the URLs for MySite portals, Following limits, UPA sync configuration (e.g. whether to sync inactive users), and Newsfeed settings, among many others. Details such as the expiration time span (TTL) and max size for these caches can be viewed with the following PowerShell commands:

PS:> $UPAProxy = Get-SPServiceApplicationProxy | ? TypeName -Match 'User Profile Service Application Proxy'
PS:> $UPAProxy.Parent

The UPA proxy configuration settings caches are examples of caches meant to save the latency of network trips. After retrieving these properties from the associated UPA once, the UPA Proxy continues to rely on them for a period of time instead of requesting them again.

With UPA configuration details, being aware of the potential for stale data is important. For example, by default the Application Properties cache, which stores key configuration details for the UPA, only refreshes every 5 minutes. So if, for example, you configure your User Profile Sync job to sync inactive as well as active users (via stsadm –o sync –ignoreisactive), the setting may not be reflected in the cached properties for up to 5 minutes. If you have encountered situations where “IgnoreIsActive” didn’t seem to be having an effect, perhaps you ran sync too soon after changing the property.

This concludes our presentation of three typical caches and how they’re used in SharePoint. In the following table, I’ve also highlighted where the cached data is stored. We’ll focus on this aspect of caches next.

Cache Data Storage Location

By caching and reusing data, the consuming service subsequently saves the resources otherwise required to produce or retrieve that data again. But the scope of those savings depends on where the cached data is stored and what other elements of the service can access it. In each of the examples in the previous section, the scope of the cache is not service-wide. Not every element of the consuming service has access to the cached data; that is, not every process in the SharePoint farm has equal access to cached items. For the BLOB cache, only processes running on the same server that originally retrieved the BLOBs can access the cache; other servers still need to make a full SQL request and acquire the authoritative data. For the page output cache, each web process instance needs to generate the page once before being able to serve it, and cannot rely on the output page cache from other processes. And for the UPA settings cache, each web or service process which instantiates the UPA proxy needs to refill its own process-local caches.

The impact and resource savings of caching data can be scaled by expanding the scope of access to the cached data. If the processed ASP.NET page were accessible to all web worker processes in the farm, for example, many additional processing runs would be avoided, saving CPU cycles. In addition, if a single processed and cached copy of the page could be made easily accessible across the farm, memory needs would be reduced as well. Since in fact the output cache is per process, each process must process the request at least once (more CPU cycles required), and each process must allocate its own memory for the output page (more RAM required).

To address the need for wider scope and centralized storage of caches and cached items, centralized cache services are deployed. Instead of individual servers and processes retrieving and storing cached items locally and internally, they store them (and sometimes even retrieve them) via a centralized cache service. As a result, other servers and processes in the environment can access and benefit from the same cached data. To increase the scalability, availability, and redundancy of these cache services, they are usually deployed as distributed services, where multiple servers and processes present a single unified interface for storing and retrieving cached data. Sound familiar? We’ve just described the role and purpose of the AppFabric Caching Service, or in SharePoint-specific parlance, the Distributed Cache Service.

To state it again explicitly, the role of the AppFabric Caching Service (in SharePoint: the Distributed Cache Service) is to provide a centralized, widely-accessible service for storage of cached data items. This central service improves performance and scalability of many SharePoint processes by multiplying the benefits of caching across all of SharePoint’s servers and processes.

We’ll clarify the role of a centralized, distributed cache service by describing the data stored in some of SharePoint’s caches.

Distributed Caches in SharePoint

You can get a list of the distributed cache types used in SharePoint by running the following PowerShell command:

[Enum]::GetNames( [Microsoft.SharePoint.DistributedCaching. Utilities.SPDistributedCacheContainerType])

The caches available as of this writing (March 2013) are listed in the following table. We’ll focus here on just two of them for now: the LogonToken and ViewState caches.

Logon Token Cache

The LogonToken cache is the one you’re likely to encounter first both practically, since it’s used as soon as you hit your first SharePoint site, and in preparing for SharePoint deployment, since it obviates the need for affinity (sticky sessions) for WFE requests.

The process of logging on to SharePoint always culminates in creation of a session token which contains key details about the user, such as his or her username. Whether initial authentication uses Windows auth, Forms auth, or a SAML token issued by an external identity provider, the final result is this session token. To avoid the multiple steps involved in getting to this token, after it’s created during initial logon, it’s cached. Prior to SharePoint 2013, this cache was a server-local cache.

Having read the previous sections, you’ll know that a cache which is stored locally to each server has the disadvantage of requiring the authentication process to be repeated again at other servers. Depending on the situation, this could result in a) a small performance hit, as in regeneration of a Windows-based token; b) more evident delays, as in the WS-Federation redirects for external identity providers; or c) additional user prompts and interference, such as requiring the user to choose an identity provider and provide credentials again. To get around these issues, a common practice was to limit sessions to interaction with a single web server, where the necessary session token was guaranteed to be cached on first logon.

In SharePoint 2013, session security tokens are stored centrally in the Distributed Cache Service’s Logon Token cache. This cache and its tokens are thereby accessible across the farm, to all WFEs. As a result, after a user initially logs on to a SharePoint farm, they can hit any web server without the problems and penalties described in the previous paragraph. The web server uses the user’s Windows identity or session cookie to find and use the session security token in the cache.

Since we’ve introduced the logon token cache here, let’s discuss an additional element relevant to it.

The Two Parts of the Logon Token Cache

The distributed logon token cache is useful, but you likely have concerns. First, if the Distributed Cache Service isn’t running properly, the logon process and all access to the farm could be completely broken. Second, even if it’s running properly, retrieving cached data from a remote cache service takes a bit longer than retrieving it from a local cache, and first logon is a time when this is particularly conspicuous.

To address these concerns, in addition to the distributed logon token cache, there’s a complementary local logon token cache on each WFE. This is also known as the “Weak Reference Cache” because of the way it stores cached tokens. As long as the weak reference cache is enabled, as it is by default, logon tokens are cached in both the distributed cache and the local weak reference cache. When cached tokens are to be retrieved, first the weak reference cache is checked, and only if nothing is found there is the distributed cache checked.

The distributed logon token cache can be cleared with the Clear-SPDistributedCacheItem –ContainerType DistributedLogonTokenCache command. The local logon token cache can only be cleared by recycling the process where it’s contained (e.g. the w3wp process).

There are two registry values which control use of the Distributed and Local (WeakRef) caches. They’re both under HKLM:\Software\Microsoft\Shared Tools\Web Server Extensions\15.0\WSS, and are both DWORDs which default to 0 (false) if not specified. They are as follows:
DisableWeakRefCacheForToken
DisableDistributedCacheForToken

Note: I’m not advocating changing these values under any circumstances in a production environment. Please contact Microsoft Global Business Support if you need assistance.

Logon Token Cache Configuration

Key properties related to the logon token caches and their default values are listed here:

PS:> Get-SPSecurityTokenServiceConfig | Format-List *LogonToken*,*WindowsToken*

MaxLogonTokenCacheItems           : 250
MaxLogonTokenOptimisticCacheItems : 100000
LogonTokenCacheExpirationWindow   : 00:10:00
WindowsTokenLifetime              : 10:00:00

MaxLogonTokenOptimisticCacheItems defines how many tokens are kept in the local Weak Reference cache, and MaxLogonTokenCacheItems specifies how many of these are stored as strong references instead of only weak references (details on weak references are available here). LogonTokenCacheExpirationWindow specifies how close to expiration tokens are considered to already be expired. So by default, tokens are considered expired 10 minutes prior to actual expiration time. Finally, WindowsTokenLifetime is used to set the Time-To-Live in the distributed logon token cache for cached security tokens of all types (not just Windows-bootstrapped tokens).

With these details in mind you should be better equipped to deploy, maintain, and troubleshoot the logon token caches. Let’s describe the usage and benefit of one other distributed cache, and then we’ll dive into details of the cache system itself.

The View State Cache

ASP.NET Web Forms maintain state between requests in a session using View State. Towards the end of the page processing lifecycle, details about the page’s state are serialized and added to the HTML to be sent to the browser as a hidden form field. When the browser posts back to the server on subsequent requests, it includes the hidden View State form field (named __VIEWSTATE). The server uses the posted back view state to rehydrate state from previously processed requests, creating the illusion of a stateful session across multiple stateless HTTP requests. ViewState has been around for a while and there are many articles available on it; here’s a good one if you’d like a quick review. A drawback of the ASP.NET ViewState system has always been the extra data transferred to the browser and back to the server on each response and subsequent request, which can reach the tens of kilobytes.

SharePoint is built on ASP.NET Web Forms and as such is dependent on ViewState and its attendant drawbacks. To improve page performance, beginning in SharePoint 2013 SharePoint caches ViewState data server-side rather than transferring it back and forth to clients. In the previously linked article, pay particular attention to the section on specifying where to persist the view state. In that section, an example is presented of caching ViewState data server side by storing it in the web server’s local file system.

SharePoint’s system is similar to that presented in the article, but such a server-specific scope would lead to some of the problems discussed previously for the old logon token cache system: subsequent requests would have to be routed to the same server where ViewState was originally cached. So instead SharePoint uses the same mechanism, but persists ViewState data to the distributed ViewState cache, sending only a GUID to the browser. Compare the __VIEWSTATE hidden form field from a SharePoint 2010 page to that from a SharePoint 2013 page, and you’ll see that only a Base64-encoded GUID remains.

ViewState data in a SharePoint 2010 page (the large field).

ViewState data in SharePoint 2013 (the last input field).

Where did all that data go? The actual ViewState data is stored in and loaded from the distributed ViewState cache, with entries for specific sessions keyed by the GUID passed back and forth from the browser. Use of this cache is governed by the SPWebService.ViewStateOnServer property, which can be accessed via these commands:

PS:> $ContentService = [Microsoft.SharePoint.Administration.SPWebService]::ContentService
PS:> $ContentService.ViewStateOnServer

These descriptions of the distributed Logon Token Cache and distributed View State Cache will help you recognize why a distributed cache service is helpful along with some of the performance and scalability advantages it presents. Now let’s move on to discuss the distributed cache service itself.

Continue with AppFabric Caching (and SharePoint) (2).

Download the scripts referenced in this post via the attachment at the end.

Replace-SPSTSSigningCertificate

Parameters

-CERPath: A location to store the newly created self-signed certificate, in case it’s needed for other applications (e.g. AD FS).
-PFXPath: A location for the new self-signed certificate together with its associated private key.
-SPSTSCertName: The name to use for the subject and issuer name on the self-signed certificate.
-RootAuthorityName: The name of the root authority to be provisioned to SharePoint.
-CleanSTSSigningCerts: A switch parameter which, if specified, cleans old certificates with the same subject name from the local Certificates store. Useful if you are testing and creating a number of certificates and want to clean up after yourself. Be careful not to delete certificates that you still need.

Dependencies

SharePoint PowerShell Snapin
New-SelfSignedCertificate and Grant-CertificateAccess (below). These are used to generate a self-signed certificate and grant access to its private key to the SharePoint farm account.

Description

Creates a new self-signed certificate with subject and issuer name as specified by the SPSTSCertName parameter. This certificate is added to SharePoint as a trusted root authority, and then the signing certificate for SharePoint’s local STS is replaced by this certificate.

After replacing the local STS signing certificate, I’ve found it useful to restart the servers in the farm. Otherwise, you may encounter authentication problems while the various processes and services learn about the new certificate.

Example

Replace-SPSTSSigningCertificate `
   -SPSTSCertName 'CN=SharePoint Security Token Service, OU=SharePoint, O=Custom, C=US' `
   -RootAuthorityName 'Self-Signed STS Signing Certificate' `
   -CerPath 'C:\temp\SPSTS.cer' `
   -PfxPath 'C:\temp\SPSTS.pfx' `
   -CleanSTSSigningCerts `
   -Verbose

New-SelfSignedCertificate

Parameters

-DN: The distinguished name to be used as both the subject and issuer of the self-signed certificate.
-DaysValid: How many days the certificate should be valid for. Defaults to 365.
-PfxPath: Where to store the PFX file associated with the certificate. PFX files contain both the certificate and the associated private key. Defaults to C:\temp\SelfSignedCert.pfx.
-CerPath: Where to store the CER file associated with the certificate. CER files contain only the certificate, not the associated private key. Defaults to C:\temp\SelfSignedCert.cer.

Dependencies

None

Description

Creates a new self-signed certificate with the specified Distinguished Name (DN). A copy of this certificate with a reference to its private key is stored in the LocalMachine\Personal store, and a PFX and CER file are stored at the designated paths.

Grant-CertificateAccess

Parameters

-UserName: The Windows account (DOMAIN\username) which should be granted access to the private key of the certificate.
-Thumbprint: The thumbprint of the certificate with private key for access.

Dependencies

Uses ICACLS.exe to set permissions on private key file.

Connect-SPFarmToAAD

Parameters

-AADDomain: The fully-qualified domain name of the Azure Active Directory tenant.
-SharePointWebUrl: The URL of the SharePoint web application to be associated with the SharePoint principal in Azure Active Directory.
-O365Credentials: A PSCredential object to use for connecting to Azure Active Directory. If this isn’t specified, you’ll be prompted.
-RemoveExistingACS: Clean up ACS proxies from the SharePoint farm.
-RemoveExistingSTS: Clean up Trusted Security Token Service objects for ACS.
-RemoveExistingAADCredentials: Clean up existing credentials associated with the SharePoint principal in AAD. Useful for keeping things tidy, but be careful not to delete credentials you need.

Dependencies

An existing Azure Active Directory tenant.
MSOL PowerShell installed and configured.
SharePoint PowerShell snapin.

Description

The function starts out by declaring a few constants and gathering credentials. Metadata about the Azure Active Directory tenant is retrieved and used to associate the OnPrem SharePoint farm with the AAD realm. The Azure ACS service application proxy is configured, as well as a Trusted Security Token Service representing AAD. Next, a connection to AAD is made (using MSOL PowerShell) and the local STS signing certificate is associated with the well-known SharePoint principal. Finally, an SPN for the hostname of the URL specified in the SharePointWebUrl parameter is added to the list of known SPNs for the SharePoint principal.

Example

Connect-SPFarmToAAD `
   -AADDomain 'joshgav1.onmicrosoft.com' `
   -SharePointWeb https://intranet.joshgav.com `
   -RemoveExistingACS `
   -RemoveExistingSTS `
   -RemoveExistingAADCredentials

Add-ServicePrincipalToAAD

Parameters

-DisplayName: A display name for the app principal in Azure AD.
-RedirectUri: Endpoint URIs which tokens should be issued for.
-HostUri: The hostname (including port number) of the endpoint. (This could be combined with RedirectUri eventually.)
-ClientId: A GUID for the app principal’s ID. Autogenerated if not specified.
-ClientSecret: A symmetric key secret for use with the client ID. Autogenerated if not specified.
-O365Credentials: Credentials for the AAD tenant.
-RemoveExistingWithSameDisplayName: Clean up existing service principals with the same display name. Careful not remove ones still in use.

Dependencies

An existing Azure Active Directory tenant.
MSOL PowerShell installed and configured.

Description

Begins with some setup: collect credentials to connect to AAD, create a client secret if one hasn’t been specified, and create an SPN to represent the location of the remote-hosted Web Application. Then the real work of connecting to AAD and creating a new service principal. The ClientSecret has to be associated with the service principal in several ways. Finally, the ClientID and ClientSecret that were used are output for further use.

Example

Add-ServicePrincipalToAAD `
   -DisplayName 'OnPremApp1' `
   -RedirectUri 'https://localhost:44300' `
   -HostUri 'localhost:44300'
   -O365Credentials $Credential `
   -RemoveExistingWithSameDisplayName

Register-SPAppPrincipalEx

Parameters

-Web: A SharePoint web. This helps set the context for the operation; the app principal is available for use across the entire SharePoint tenancy.
-ClientId: The client ID (a GUID) for this app principal.
-ClientSecret: The Base64-encoded secret for this app principal.
-DisplayName: Display name for the app principal.
-HostUri: The hostname (including port) where the remote-hosted app is published.
-RedirectUri: The HTTP/S endpoint where requests for apps associated with this app principal will be redirected.

Dependencies

SharePoint PowerShell snapin.

Description

Using the specified parameters, creates an AppPrincipal in SharePoint to be associated with apps. Note the similarity between this function and the one for adding a service principal to AAD. Eventually, app principal registrations in SharePoint will automatically create a parallel registration in AAD. In the meantime, specify the same values for each function.

Example

$AppPrincipal = Register-SPAppPrincipalEx `
   -Web https://intranet.joshgav.com `
   -ClientId $ClientInfo.ClientId `
   -ClientSecret $ClientInfo.ClientSecret `
   -HostUri 'localhost:44300' `
   -RedirectUri 'https://localhost:44300' `
   -DisplayName $ClientInfo.ClientDisplayName

Configure-SPFarmAppIsolation

Parameters

-AppPrefix: The prefix placed before every AppWeb URL.
-AppDomain: The domain which follows the AppWeb prefix.
-ADDomain: The AD Domain in which the wildcard DNS entry for the AppDomain will be placed.
-CanonicalName: The server to which the CNAME record for the wildcard DNS entry should point.
-AddWildcardDNSEntry: Specify $false to not add the DNS entry. Defaults to true; use -AddWildcardDNDEntry:$false to specify false.

Dependencies

SharePoint PowerShell snapin

Description

Sets up App Isolation for SharePoint.

Example

Configure-SPFarmAppIsolation `
   -AppPrefix 'apps' `
   -AppDomain 'apps.intranet.joshgav.com' `
   -ADDomain 'gavant.local' `
   -CanonicalName 'intranet.joshgav.com' `
   -AddWildcardDNSEntry

LowTrustApps.zip

Note: I’ve been wanting to share the details on setting up an OnPrem SharePoint environment to support low-trust apps for a while now. I had some help from the Product Group to get things working, and there were several catches and ‘Aha!’ moments involved which many of you are I’m sure stumbling over. So I dug in today and hammered out this post and the associated post of PowerShell helper functions to get this info out there. Let me know if (and when!) you find details that need clarification or might need correction. Enjoy!

A quick summary of SharePoint apps

SharePoint apps can be hosted inside SharePoint or outside of it. Apps hosted outside of SharePoint can be manually provisioned to an IIS server, an Apache server, an Azure Compute Services role, or any other host; or can be automatically provisioned and hosted in Azure Web Sites. Externally-hosted apps can call into SharePoint to request data on behalf of the user using the App Authentication model, an implementation of OAuth. Of the several OAuth profiles available, SharePoint apps implement and extend two of them under the names “High-Trust” and “Low-Trust.” In this post, we’ll discuss the protocol flow for SharePoint low-trust apps and how to configure an On-Prem SharePoint farm to support it.

A brief history of OAuth

OAuth version 1 was prepared by some of the biggest names of Web 2.0 - Twitter, Google, Facebook, Flickr. It provides a simple means for users to delegate limited authorization to their data to applications and services without sharing master credentials (e.g. username and password). For example, by using OAuth Facebook doesn’t need to ask you to share your Google password to get your Google contacts; instead you delegate to Facebook specific authorization to read your Google contacts list. In the canonical example (from the RFC), you can grant a photo printing service authorization to read your photos from a photo storage service without sharing your password with the printing service or granting that service read/write privileges to your photos and other account data at the photo storage service provider.

OAuth version 2 has been under discussion for several years now and RFC 6749 is the first of several RFCs documenting it. Significant (and controversial) additions to OAuth 2 include the JavaScript Web Token (JWT, pronounced ‘jot’) profile and the OAuth Assertion Framework, which together have made it easier to use OAuth for authentication as well as delegated authorization. Microsoft’s implementations of OAuth make heavy use of JWTs, and you’ll find the JWT Bearer Token profile to be the foundation for Microsoft’s High-Trust, Server-to-Server authentication schemes. For low-trust authentication, SharePoint uses a version of the classic authorization code flow discussed in the previous paragraph.

OAuth in SharePoint low-trust apps

SharePoint low-trust apps rely on the OAuth authorization code flow (“grant type”) to delegate limited rights to apps to act as users. For this to work, both SharePoint and the Client application (the SharePoint app) must trust and communicate with an Authentication Provider; SharePoint relies on Azure Active Directory. Azure AD, in turn, must be aware of SharePoint and the Client app in order to grant them the necessary codes and tokens to work together.

Let’s orient ourselves by presenting the overall protocol flow:

In this picture, the ‘S’ icon represents SharePoint and the globe represents an externally-hosted SharePoint app. The floating blue window represents Azure Active Directory, as indicated. In step 1, a user requests a page or an app directly from SharePoint. The app could be hosted directly within a SharePoint page in an IFrame, or might be on its own page. In either case, SharePoint will need to send the user’s browser to the external host to retrieve the app page, either via the location tag of the IFrame element or a 302 Redirect HTTP status; it will do so in step 3. First, though, SharePoint gets an OAuth authorization code to share with the app; the app will use this code to ultimately communicate back with SharePoint acting as the user. In step 2, SharePoint asks Azure Active Directory Access Control Service for an authorization code for the app. Now, in step 3, SharePoint redirects the user’s browser to the app, including the authz code with the redirected request (as the SPAppToken form parameter).

The app doesn’t use the authorization code directly to authenticate to SharePoint. Instead, in step 4 it calls Azure AD ACS and exchanges the authorization code it received from SharePoint for a proper Access Token. In step 5 the app uses this access token to authenticate to and access data from SharePoint, and finally in step 6 it uses this data to present a page to the user.

Setup SharePoint for low-trust apps

There’s lots to say about the various steps in this flow, but our focus here will be on setting up an On-Premises SharePoint 2013 farm to participate in it via a connection to Azure Active Directory. Most of this post will focus on enabling step 2 above - the connection between SharePoint and Azure Active Directory. The three steps you’ll need to follow to setup a SharePoint farm for low-trust apps are as follows:

1. General SharePoint setup for apps (e.g. App Management Service Application, app isolation).
2. Connect SharePoint to Azure Active Directory.
3. Create an App Principal in AAD and SharePoint.

Step 1 is already well-documented on TechNet and other Web sites, so we’ll start at step 2.

Connect SharePoint to Azure Active Directory

If you haven’t yet, follow the steps in my previous post to get the Azure Active Directory module for PowerShell set up. You’ll need an AAD tenant and AAD/MSOL PowerShell for the next steps, and you’ll follow the discussion better after you’ve experimented a bit with AAD.

To completely configure a connection between your OnPrem SharePoint farm and Azure AD, you’ll need to complete the following steps:

1. Replace the local STS signing certificate with one Azure AD can trust. I use a self-signed certificate, though a certificate from a trusted global authority should also work.
2. Associate this certificate with the SharePoint principal in Azure Active Directory.
3. Create an SPN for the OnPrem SharePoint farm and add it to the SharePoint principal in Azure Active Directory.
4. Configure the authentication realm for the local SharePoint farm to match the AAD realm.
5. Create an Azure Access Control Service application proxy in SharePoint.
6. Create a Trusted Security Token Service for Azure ACS in SharePoint.

Remember that in step 2 above SharePoint calls in to AAD to request an authorization code, which requires authentication. The SharePoint username for this authentication is a well-known GUID representing SharePoint in Azure AD and O365 (00000003-0000-0ff1-ce00-000000000000). The credential used is an X.509 certificate, specifically SharePoint’s local STS signing certificate. Worth stating again: the same certificate used by SharePoint’s local STS to sign security tokens is used by SharePoint to authenticate to Azure Active Directory. That’s why the first step above is to replace the local signing certificate with one that can be used for authentication to AAD

I’ve created PowerShell functions which take care of the above steps and which you can review for the technical nitty-gritty; they’re named Replace-SPSTSSigningCertificate and Connect-SPFarmToAAD. You can access them individually at this post’s sister post, or download the whole package of scripts related to this post here. For further details, refer to the sister post and to the scripts themselves.

If you’ve saved all the files to the same directory, you can navigate there and use the following commands to set up your connection, replacing the AADDomain name and SharePoint Web URL appropriately. In the zip file, a similar script is available named ‘_ConnectSharePointToAAD.ps1’.

$FolderPath = if ($psISE) {Split-Path -Path $psISE.CurrentFile.FullPath -Parent} elseif ($PSScriptRoot) {$PSScriptRoot} else {"."}
Import-Module -Name ('{0}\{1}' -f $FolderPath,'New-SelfSignedCertificate.ps1')
Import-Module -Name ('{0}\{1}' -f $FolderPath,'Grant-CertificateAccess.ps1')
Import-Module -Name ('{0}\{1}' -f $FolderPath,'Replace-SPSTSSigningCertificate.ps1')
Import-Module -Name ('{0}\{1}' -f $FolderPath,'Connect-SPFarmToAAD.ps1')

Add-PSSnapin Microsoft.SharePoint.PowerShell
Import-Module -Name MSOnline

Replace-SPSTSSigningCertificate `
   -SPSTSCertName 'CN=SharePoint Security Token Service, OU=SharePoint, O=Custom, C=US' `
   -RootAuthorityName 'Self-Signed STS Signing Certificate' `
   -CerPath 'C:\temp\SPSTS.cer' `
   -PfxPath 'C:\temp\SPSTS.pfx' `
   -CleanSTSSigningCerts `
   -Verbose

Write-Warning 'The STS Signing Certificate has been replaced. It is recommended that you restart your system before continuing.'

Connect-SPFarmToAAD `
   -AADDomain 'joshgav1.onmicrosoft.com' `
   -SharePointWebUrl https://intranet.joshgav.com `
   -RemoveExistingACS `
   -RemoveExistingSTS `
   -RemoveExistingAADCredentials `
   -Verbose

Create App Principals

We’ve connected SharePoint to AAD, but to actually utilize this connection, we need to create AppPrincipals in Azure Active Directory and SharePoint. These AppPrincipals represent the remote-hosted Web Applications which will connect to SharePoint acting as users. Azure AD needs to be aware of these principals to be able to issue authorization codes for them (step 2) and access tokens to them (step 4). SharePoint needs to be aware of them to allow them access on behalf of users (step 5).

As things currently stand, you must create principals separately in SharePoint and in AAD. Work is underway to allow SharePoint to create a service principal in AAD automatically. In the meantime, though, we’ll use two separate functions to create an AppPrincipal in AAD and SharePoint. Again, refer to the sister post for details on these two functions, Add-ServicePrincipalToAAD and Register-SPAppPrincipalEx. Note that you’ll need the ClientID and Client Secret you use here when you create your SharePoint app, so don’t lose them!

An example, included in the zip as ‘_NewAppPrincipal.ps1,’ follows:

$FolderPath = if ($psISE) {Split-Path -Path $psISE.CurrentFile.FullPath -Parent} elseif ($PSScriptRoot) {$PSScriptRoot} else {"."}
Import-Module -Name ('{0}\{1}' -f $FolderPath,'Register-SPAppPrincipalEx.ps1')
Import-Module -Name ('{0}\{1}' -f $FolderPath,'Add-ServicePrincipalToAAD.ps1')

$Credential = Get-Credential -Message 'O365 Credential'

Add-PSSnapin Microsoft.SharePoint.PowerShell
Import-Module MSOnline

$ClientInfo = Add-ServicePrincipalToAAD `
   -DisplayName 'OnPremApp1' `
   -RedirectUri 'https://localhost:44300' `
   -HostUri 'localhost:44300' `
   -O365Credentials $Credential `
   -RemoveExistingWithSameDisplayName

$AppPrincipal = Register-SPAppPrincipalEx `
   -Web https://intranet.joshgav.com `
   -ClientId $ClientInfo.ClientId `
   -ClientSecret $ClientInfo.ClientSecret `
   -HostUri 'localhost:44300' `
   -RedirectUri 'https://localhost:44300' `
   -DisplayName $ClientInfo.ClientDisplayName

$ClientInfo

Create a Remote-Hosted App

At this point, your environment is set. You can create a provider-hosted app in Visual Studio and specify the ClientID and ClientSecret created here in the web.config file. The MSDN samples have a Hello World app to get you started. Don’t forget to change the ClientID and ClientSecret in web.config.

Updated 2/28/2013 to account for the updated AAD PowerShell module (https://aka.ms/aadposh).

Sign up for Azure Active Directory

Azure Active Directory provides the directory and access control service for Office 365, so if you have an O365 subscription of any sort you already have a corresponding AAD tenancy. If you don’t have one or prefer a different tenancy for testing, sign up for an AAD account here, or a new O365 Enterprise Preview account here.

You can access a Web UI for user management at https://activedirectory.windowsazure.com. Right now the only significant task you can accomplish via the UI is some user management. Managing service principals and identity and authentication configuration requires PowerShell.

Install and Prepare MSOL PowerShell

Azure Active Directory PowerShell was formerly known as Microsoft Online (MSOL) PowerShell, and most cmdlets still use the MSOL prefix. The module requires the Microsoft Online Sign-In Assistant (MSO SIA). You can download the Sign-In Assistant here and the 64-bit PowerShell module here (reference to 32-bit and 64-bit editions of the SIA and AAD module are here).

Note (Feb 28, 2013): The following workaround is no longer necessary with the release of the AAD PowerShell module. You can still delete the MSOnlineExtended folder if you want, but the CmdletsToExport line is now correct by default.

Before you get started with your new cmdlets, you’ll need to work around some problems. AAD PowerShell installs two modules into the PowerShell module directory at C:\windows\system32\windowspowershell\v1.0\modules. These are the MSOnline and MSOnlineExtended modules. At this point, both modules include the exact same DLL; as a result, PowerShell can’t load both modules in the same session at once. The best way to deal with this is as follows:

Throw out the entire MSOnlineExtended module. If it makes you nervous, just move it somewhere else. But I promise you won’t have any issues if you completely delete it. (Note: This is current as of Oct 23, 2012).

Next, open up the MSOnline module folder (in the $pshome\modules directory), and open MSOnline.psd1 somewhere you can edit it. One option is to run this command in the PowerShell ISE (running as Administrator):

PS C:\> psedit $pshome\modules\msonline\msonline.psd1

Find the line for CmdletsToExport and change it to this:

PS C:\> CmdletsToExport = ‘*’

What you’re doing here is specifying that all available cmdlets in MSOL PowerShell DLLs should be exported into your PowerShell sessions via the MSOnline module.

Once you’ve done this, re-open PowerShell and run:

PS C:\> Get-Command -Module MSOnline

If all is well, you’ll see a list of all the Azure AD cmdlets. Look for the MsolServicePrincipal ones in particular.

Connect to AAD

Now we’ll get acquainted with the service principals which initially populate your directory. First, you’ll need to establish the context for your session via the Connect-MsolService cmdlet. Just enter the cmdlet and put in your administrative credentials for your O365/AAD tenancy. In the background, a persistent connection is set up and cached for use with all the MSOL cmdlets you use in your session.

Curious what’s happening behind the scenes here? Open up a Web Proxy Debugger (e.g. Fiddler2), make sure you have set it to decrypt HTTPS connections, and then run the Connect-MsolService cmdlet. You’ll see that a special SOAP request is sent to the BEC Web Service. BEC, or Business Online Services Engineering Core, is Microsoft’s name for the gateway to Azure AD. You’ll occasionally see it turn up in contexts where a connection to AAD is needed, such as in SharePoint 2013 PowerShell (New-SPBecWebServiceApplicationProxy).

Service Principals and Domains

The first cmdlet to run once you’re all connected is Get-MsolServicePrincipal. If you have O365 set up, you’ll see principals for each of the online services, such as Exchange, SharePoint, and Lync. Here’s a list of the principals currently in my AAD domain.

PS C:\> Get-MsolServicePrincipal | ft DisplayName, AppPrincipalId -AutoSize

DisplayName                         AppPrincipalId
-----------                         --------------
Microsoft.Lync                      00000004-0000-0ff1-ce00-000000000000
Microsoft.Office365Portal           00000006-0000-0ff1-ce00-000000000000
Microsoft.SharePoint                00000003-0000-0ff1-ce00-000000000000
Microsoft.Azure.Workflow            00000005-0000-0000-c000-000000000000
Microsoft.Azure.AnalysisServices    00000009-0000-0000-c000-000000000000
Microsoft.Exchange                  00000002-0000-0ff1-ce00-000000000000
Microsoft.ExchangeOnlineProtection  00000007-0000-0ff1-ce00-000000000000
Microsoft.Azure.ActiveDirectory     00000002-0000-0000-c000-000000000000
“Napa” Office 365 Development Tools 48717084-a59c-4306-9dc4-3f618dbecdf9
OnPremApp1                          52f3de70-2d75-4187-9c49-138d86e54244
rp.joshgav.com                      66b3011d-892c-4cbe-a844-61d87ecdf8f8

Notice the last 3? Those are principals that I’ve added as relying OAuth and WS-Trust parties. The last one is a Relying Party Web Application of mine. I created the principal in AAD and then ran the Identity and Access tool in Visual Studio to federate with Azure AD for single-sign on. More on that another time.

If Service Principals are approximately equivalent to Relying Parties (WS-Federation) and Service Providers (SAML), then Domains are Identity Providers (WS-Fed) and Authentication Authorities (SAML). Run Get-MsolDomain to see all of your AAD domains. If you’ve federated your O365 directory with an On-Prem ADFS STS, you’ll see it listed as a Federated domain here. I’ve federated ‘gavant.org.’

PS C:\> Get-MsolDomain | ft -AutoSize

Name                          Status   Authentication
----                          ------   --------------
gavant.org                    Verified Federated
joshgav1.onmicrosoft.com      Verified Managed 

If you have a federated domain, run Get-MsolDomainFederationSettings to see how AAD routes authentication requests back to your OnPrem ADFS instance. Here’s my configuration:

PS C:\> Get-MsolDomainFederationSettings -DomainName gavant.org

ActiveLogOnUri         : https://fs.joshgav.com/adfs/services/trust/2005/usernamemixed
FederationBrandName    : JoshGav ADFS
IssuerUri              : https://fs.joshgav.com/adfs/services/trust/
LogOffUri              : https://fs.joshgav.com/adfs/ls/
MetadataExchangeUri    : https://fs.joshgav.com/adfs/services/trust/mex
NextSigningCertificate :
PassiveLogOnUri        : https://fs.joshgav.com/adfs/ls/
SigningCertificate     : MIIC2DCCAcCgAwIBAgIQf2RJ+RmnbqhMANBCDJz6ojANBgkqhkiG9w0BAQsFADAoMSYwJAYDVQQDEx1BREZTIFNpZ25pbmcgLSBmcy5qb3NoZ2F2LmNvbTAeFw0xMjEwMDExMzI4MjhaFw0xMzEwMDExMzI4MjhaMCgxJjAkBgNVBAMTHUFERlMgU2lnbmluZyAtIGZzLmpvc2hnYXYuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAuZ6zVYGdqKk04lCxbqjZjd+Nhu7GlFhnz1RBWKNevxWPP9bsMR5B/...

More Metadata

We’ve briefly seen Service Principals and Identity Providers. What we’re missing is information about the AAD STS itself. The best way to see more about this is to retrieve the Federation Metadata document and the JSON Metadata document for your AAD realm. Assuming {0} to be a placeholder for your AAD domain name or realm ID, the following URLs will get you the information you need.

I have no idea what standard the JSON information adheres to, but Microsoft uses it here and in other products.

For example, you can retrieve the metadata for my AAD tenant at https://accounts.accesscontrol.windows.net/joshgav1.onmicrosoft.com/FederationMetadata/2007-06/FederationMetadata.xml. Go ahead, click it.

Personally, I prefer to use PowerShell:

PS:> $AADDomain = ‘joshgav1.onmicrosoft.com’
PS:> [xml] $Output = (
    Invoke-RestMethod `
        -Uri (‘https://accounts.accesscontrol.windows.net/{0}/FederationMetadata/2007-06/FederationMetadata.xml’ -f $AADDomain)).SubString(1)

PS:> Invoke-RestMethod -Uri (‘https://accounts.accesscontrol.windows.net/{0}/metadata/json/1’ -f $AADDomain)

The metadata document returned in the first command has an extra character at the beginning, which is why we have the extra SubString(1) method tacked on. Invoke-RestMethod is a very helpful PowerShell cmdlet which converts JSON objects into PowerShell objects automatically, as you can in the last command above.

The metadata you receive is all about the Azure AD Access Control Service. You’ll notice it has a Principal ID of 00000001-0000-0000-c000-000000000000 (which was missing in the list above). The perceptive amongst you will notice that its main endpoint is https://accounts.accesscontrol.windows.net/RealmID, which may remind you of the ACS 2.0 domains. In ACS 2.0, domains are named https://<namespace>.accesscontrol.windows.net. In AAD ACS, the root domain accounts.accesscontrol.windows.net is shared by all tenants.

Summary

In this post we’ve gotten you set up with an AAD account, installed and fixed MSOL PowerShell, and taken a brief tour of AAD service principals, domains, and the AAD Access Control Service itself. Use this knowledge to begin composing and federating domains and identity providers with your AAD tenancy.

Invoke-SecurityTokenRequest.zip

01 - SharePoint, Claims, WS-* and the Windows Identity Foundation (WIF)
02 - WS-Federation Authentication Module and SharePoint Extensions
03 - The SharePoint Federation Authentication Module
04 - Integrating SharePoint with Windows Azure Active Directory (ACS)
05 - Session Authentication Module and WS-SecureConversation
06 - SharePoint Session Authentication Module
07 - Windows Claims Authentication
08 - Service (ActAs) Tokens

Thank you to Steve Peschka for reviewing these articles!

SharePoint Claims Authentication Part 3

Cool as WIF is, this series is meant to help with understanding SharePoint authentication in particular. So in this article we will build on the foundation laid in the first two and describe how the SharePoint Federation Authentication Module complements and extends WIF's WSFAM.

For the sake of being complete, I'll mention here that the first thing the SPFAM does is hook up SharePoint's custom certificate validation process, so that root certificates are checked against the collection stored in SharePoint in addition to the local machine store. SharePoint has its own built-in Certificate Authority; you can retrieve a list of trusted root certificates via Get-SPTrustedRootAuthority and add new ones with Add-SPTrustedRootAuthority.

Moving on, the two points where SPFAM diverges significantly from its WSFAM roots are: 1) in the process of initial redirection of the web browser when it first contacts a SharePoint web application; and 2) in the process of setting up the Session Security Token after authentication has completed in the second communication between browser and web application (after the browser is redirected back to the application). We'll focus in this article on how SharePoint handles initial redirection to an STS and return to session tokens later. It is important to emphasize that if Windows Claims and only Windows Claims is enabled for a Web Application, a different process is followed which we will cover later in its own article.

First Redirections

Assuming Windows Claims is not enabled, SharePoint starts by causing two sequential redirections to take place. As with all the interactions described above and below, you can follow along at home in Fiddler. The first two redirections are to:

1. /_layouts/Authenticate.aspx
2. /_login/default.aspx

The redirection to Authenticate.aspx leads to a page which tries one last time to authenticate the incoming user. I'm not sure why this is here, but perhaps we'll figure it out as we continue or perhaps a reader will know and tip us off. I believe it may help with authentication on the return trip, where our browser will visit this page again. The second redirection is initiatied by the FormsAuthenticationModule based on the following entry in web.config:

<authentication mode="Forms">
  <forms loginUrl="/_login/default.aspx" />
</authentication>

Home Realm Discovery

At /_login/default.aspx the fun continues! This page is known as the MultiLogonPage and is responsible for a process called Home Realm discovery. We've referred to the URL of the SharePoint (or other) web application as the Trusting Realm. A user's Home Realm is the realm and associated STS which have first-hand knowledge of the user and are willing and capable of ascertaining and attesting to the user's identity. For example, as a Microsoft employee my home realm is generally microsoft.com. Determining a user's home realm is not always a trivial process and unfortunately sometimes requires direct user interaction. That is the case in SharePoint if more than one authentication provider has been specified for a Web Application, and is the purpose of the MultiLogon page. If more than one authentication provider has been configured, the user will be presented here with a drop-down box from which to choose their home realm - the realm which is capable of authenticating him or her.

As an aside, other options have been proposed for home realm discovery in SharePoint, such as those documented by Steve Peschka here and here and Chris Keyser here.

Luckily, in situations where only one authentication provider has been configured, the MultiLogonPage is smart enough to continue the process of sending the user to the proper STS without having to bother them further. In either case (that is, whether the STS is determined automatically or the user chooses a realm and associated STS from the dropdown), the next step is to redirect the user to the AuthenticationRedirectionUrl associated with that STS. To review these redirection URLs, you can run Get-SPAuthenticationProvider in PowerShell and check the AuthenticationRedirectionUrl property. As you will see, this next redirection is again within the SharePoint application and is the last one before we leave for our STS. In a Web Application of mine configured with a single authentication provider, here's the output of that command and the next step in our journey:

PS:> Get-SPAuthenticationProvider -WebApplication https://intranet.gavant.local -Zone Default

DisplayName                  : AzureACS
LoginProviderName            : AzureACS
ClaimProviderName            :
AuthenticationRedirectionUrl : /_trust/default.aspx?trust=AzureACS
UpgradedPersistedProperties  :

And yes, before all is said and done we will walk through federating a SharePoint web application with Azure ACS. Let's explore what happens in this step. The page (_trust/default.aspx) and it's web.config are stored at 14\TEMPLATE\IDENTITYMODEL\TRUST.

Off to the STS

The key logic on this page checks the "trust" parameter of the incoming query string and prepares to finally redirect the browser to the actual STS associated with the named Trusted Provider. Note that the name of my Trusted Provider is AzureACS. If I run Get-SPTrustedIdentityTokenIssuer, I retrieve all information needed to send a WS-Federation request to that trusted provider, as shown here:

PS:> Get-SPTrustedIdentityTokenIssuer

ProviderUri                       : https://joshgav.accesscontrol.windows.net/v2/wsfederation
DefaultProviderRealm              : https://intranet.gavant.local
ProviderRealms                    : {}
ClaimTypes                        : {http://schemas.xmlsoap.org/ws/2005/05/identity/claims/nameidentifier, http://schemas.microsoft.com/accesscontrolservice/2010/07/claims/IdentityProvider}
HasClaimTypeInformation           : True
ClaimTypeInformation              : {ACSNameIdentifier, ACSIdentityProvider}
IdentityClaimTypeInformation      : Microsoft.SharePoint.Administration.Claims.SPTrustedClaimTypeInformation
ClaimProviderName                 :
UseWReplyParameter                : False
Description                       : AzureACS
SigningCertificate                :  [Subject]
                                      CN=joshgav.accesscontrol.windows.net

  [Issuer]
   CN=joshgav-ca, DC=joshgav, DC=com

  [Serial Number]
   2F3422D100000000000A

  [Not Before]
   3/28/2012 10:59:41 AM

  [Not After]
   3/28/2014 10:59:41 AM

  [Thumbprint]
   F8EA661189C2B1B6098B44EEB9427B2969A4FA23

Name                             : AzureACS
TypeName                         : Microsoft.SharePoint.Administration.Claims.SPTrustedLoginProvider
DisplayName                      : AzureACS
Id                               : 24c187f3-22cb-4bb9-a45c-fc31a786a1ad
Status                           : Online
Parent                           : SPSecurityTokenServiceManager Name=SecurityTokenServiceManager
Version                          : 3381307
Properties                       : {}
Farm                             : SPFarm Name=SPS2010RTM_SharePoint_Configuration_Database
UpgradedPersistedProperties      : {}

And using this information, that's exactly what is done! This info is built into the proper WS-Federation URL with query string and my browser is redirected to the appropriate STS where the next stage in the WS-Federation profile can take place. After that is complete, the RSTR is posted back to this same location (/_trust/) and the AuthenticateRequest stage of the standard WSFAM is invoked as described before to complete the authentication process. The full sequence of HTTP requests (copied from Fiddler) is shown here (click for a larger view):

Note that in the two messages where a 200 is received followed by a POST in the next message, most details have been included in a form which is posted to the next hop. In the other GET requests information is sent to the next hop in the query string.

Conclusion

Congratulations! If you've come this far you are pretty well equipped to understand the Claims logon process in a SharePoint web application. There is still quite a ways to go, as we have yet to discuss Session and Security Context Tokens, the special case of Windows claims, and how claims works for service applications, but there is always another day J. In my next post, we'll take time for some fun practical application and I'll walk you through federating a SharePoint web application with Azure Access Control Service with the usual commentary along the way. As a bonus, we'll do it completely via PowerShell. In the meantime, rest your head!

01 - SharePoint, Claims, WS-* and the Windows Identity Foundation (WIF)
02 - WS-Federation Authentication Module and SharePoint Extensions
03 - The SharePoint Federation Authentication Module
04 - Integrating SharePoint with Windows Azure Active Directory (ACS)
05 - Session Authentication Module and WS-SecureConversation
06 - SharePoint Session Authentication Module
07 - Windows Claims Authentication
08 - Service (ActAs) Tokens

Thank you to Steve Peschka for reviewing these articles!

SharePoint Claims Authentication Part 2

In the Windows Identity Foundation (WIF) and Claims web applications utilizing it, the WS-Federation Authentication Module (WSFAM) and Session Authentication Module (SAM) are responsible for authentication. Like existing authentication modules for IIS and ASP.NET such as the Windows Authentication and Forms Authentication modules, the WIF authentication modules plug into stages in the integrated pipeline to interact with and influence request processing.

SharePoint implementation of WSFAM and SAM

As in so many other areas, SharePoint wraps and simplifies management of Claims authentication via WIF. It does this by deriving custom classes from the standard WSFAM and SAM authentication modules: the SPFederationAuthenticationModule and SPSessionAuthenticationModule, respectively. SharePoint also adds a third module to the pipeline, SPWindowsClaimsAuthenticationHttpModule, which deals specifically with creating Claims-based tokens for Windows users. More on that later. The extensions to the web.config files when Claims authentication is enabled are contained within 14\CONFIG\IdentityModel\webconfig.identitymodel.add.xml; the new modules appear like this in a Web Application's web.config file:

<add name="FederatedAuthentication" type="Microsoft.SharePoint.IdentityModel.SPFederationAuthenticationModule, Microsoft.SharePoint.IdentityModel, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" />
<add name="SessionAuthentication" type="Microsoft.SharePoint.IdentityModel.SPSessionAuthenticationModule, Microsoft.SharePoint.IdentityModel, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" />
<add name="SPWindowsClaimsAuthentication" type="Microsoft.SharePoint.IdentityModel.SPWindowsClaimsAuthenticationHttpModule, Microsoft.SharePoint.IdentityModel, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" />

Some settings for Claims authentication can be configured declaratively in web.config files like some settings of the Forms, Windows, and other authentication modules. The <microsoft.identityModel> section is used to declare these settings and is added as a direct child of the <configuration> element. In SharePoint web.config files it tends to be at the very bottom of the file. The default configuration for SharePoint web applications is provided in the same XML file as the additional module configuration described above and is provided below for reference. I'd highly recommend not changing this without clear approval or instruction from Microsoft Support. Some of these settings may be surprising, such as passiveRedirectEnabled="false", but as we shall see their values have been chosen for a reason.

<microsoft.identityModel>
   <service saveBootstrapTokens="true">
      <audienceUris />
      <issuerNameRegistry type="Microsoft.SharePoint.IdentityModel.SPPassiveIssuerNameRegistry, Microsoft.SharePoint, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" />
      <securityTokenHandlers>
         <clear />
         <add type="Microsoft.IdentityModel.Tokens.X509SecurityTokenHandler, Microsoft.IdentityModel, Version=3.5.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
         <add type="Microsoft.SharePoint.IdentityModel.SPSaml11SecurityTokenHandler, Microsoft.SharePoint.IdentityModel, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c">
            <samlSecurityTokenRequirement>
               <nameClaimType value="http://schemas.microsoft.com/sharepoint/2009/08/claims/userid" />
            </samlSecurityTokenRequirement>
         </add>
         <add type="Microsoft.SharePoint.IdentityModel.SPTokenCache, Microsoft.SharePoint.IdentityModel, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" />
      </securityTokenHandlers>
      <federatedAuthentication>
         <wsFederation passiveRedirectEnabled="false" issuer="https://none" realm="https://none" />
         <cookieHandler mode="Custom" path="/">
            <customCookieHandler type="Microsoft.SharePoint.IdentityModel.SPChunkedCookieHandler, Microsoft.SharePoint.IdentityModel, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" />
         </cookieHandler>
      </federatedAuthentication>
    </service>
</microsoft.identityModel>

Of the various stages in the ASP.NET pipeline (see the Remarks section at the end of the linked page), you will not be surprised to learn that the two WIF authentication modules register for the AuthenticateRequest and PostAuthenticateRequest events. You may be surprised that the WSFAM module registers for the EndRequest event as well. We'll now explain how these modules work, beginning with WSFAM and SPFAM. In this post, we'll describe WSFAM independently and in the next article describe how SharePoint modifies its behavior.

WSFAM: The federation sequence

As its name implies, the WS-Federation Authentication Module is responsible for implementing the WS-Federation profile to request and receive security tokens via a web browser (passive) client. In simple WIF applications, properties such as the federated STS, the source realm, and the desired authentication type can be set in the web.config file at the path configuration/microsoft.identityModel/service/federatedAuthentication/wsFederation as detailed here. If you're familiar with the WS-Federation spec you'll immediately recognize the correspondence between query parameters specified therein and the configuration properties available for WSFAM.

web.config-based configuration of WSFAM allows federation with a single Security Token Service. This may be fine for small applications, but in SharePoint more options are needed. As a result, the wsFederation element for a SharePoint web application is filled mostly with placeholder content. Instead of using this element, SharePoint retrieves details about federation configuration and requirements from its own object model in the SPFAM module process, as we'll discuss later.

AuthenticateRequest

As mentioned above, WSFAM registers itself for the AuthenticateRequest pipeline event. Logically and intuitively, the WSFAM passes through this stage first. However, the actions taken at this stage differ significantly on the second time through the pipeline, so we will address the stages of WSFAM as follows:

1. AuthenticateRequest (first pass)
2. PostAuthenticateRequest
3. EndRequest
4. AuthenticateRequest (second pass)

As it passes through the AuthenticateRequest stage, the WSFAM checks the incoming request for a form with a wresult parameter. If you recall from part one of this series, the RSTR from the STS is returned to the Relying Party (RP) application embedded in this wresult form parameter. So in other words, this stage expects to take action only if the browser has already visited and returned from the STS with a security token. That will be the case later on, in the second pass (stage 4). However, if this is the browser's first contact with the web application, the AuthenticateRequest process in WSFAM will not find a wresult parameter and will thus have nothing to do.

PostAuthenticateRequest

In the PostAuthenticateRequest stage, WSFAM checks if a Claims principal has been properly set on the current HTTP context. If authentication succeeded in the AuthenticateRequest stage some principal will have been set on the context, though this principal could be a Windows or Forms principal and not a Claims principal. In that case, the WSFAM converts them into a Claims Principal at this stage. If no principal has been set (i.e. authentication has not yet succeeded), an anonymous Claims principal is set for the context now. In SharePoint, either a Claims principal should already have been set at this point (if authentication has completed, i.e. during the second pass) or the anonymous principal will be set.

After a proper Claims principal has been set on the context, any custom Claims Authentication Manager configured for the application is called. For SharePoint web applications, no Claims Authentication Manager is configured; however, there is a custom Claims Authentication Manager for service applications. Claims Authentication Managers can modify or augment claims set for the incoming principal, or even entirely replace that principal. This may be a valid extensibility point even in a SharePoint web application; however, a custom ClaimProvider would generally be more appropriate.

EndRequest

Finally, we arrive at the EndRequest stage, which is the most significant point on a request's first pass through the pipeline. It is at this point that the browser is redirected to an STS to request a security token. Significantly, by default this entire stage is skipped if the passiveRedirectEnabled property on the wsFederation configuration element is set to "false". As you can see in the configuration above, this is in fact the case for a SharePoint web application. SharePoint provides its own version of OnEndRequest to inject the more complex logic needed there. Let's go ahead and look at what WSFAM does on its own first.

First, EndRequest checks if the incoming request is not authorized by checking for a 401 response code. Thus, this stage is generally only relevant on the first pass through the pipeline. Of course, it's possible that a user was properly authenticated but still not authorized to access the target resource, so EndRequest also checks if an authenticated user has been set on the context. If it hasn't, redirection to the STS is initiated. This is pretty simple - parameters set declaratively and programmatically for the request to the STS are gathered together and placed in a query string, itself appended to the URL of the designated STS. This URL is sent to the browser as a 302 response and the browser heads off to the STS to request a security token, as described in Part 1.

The next step at this point in the WS-Federation profile is authentication at the STS, receipt of a security token from the STS, and posting of that token back to the RP web application. See the first post in this series for more elaboration on these details. Once the RSTR is posted back to the RP web application, we reach the AuthenticateRequest stage in the ASP.NET pipeline again, and we continue our discussion there.

AuthenticateRequest (second pass)

Coming through AuthenticateRequest after returning from the STS, the browser should now be presenting a form with a valid wresult parameter containing an RSTR (RequestSecurityTokenResponse). This time through WSFAM has a lot to do! It must read in the RSTR, find and call appropriate security token handlers, and finally pass off control to the Session Authentication Manager to create a session security token. Let's step through this process step by step.

First, the web form in the incoming request is parsed. The wresult parameter, which contains the RSTR and associated security token, is the most important component here. It is retrieved and then passed to the collection of SecurityTokenHandlers specified in the web.config file (or programmatically). SecurityTokenHandlers are elements in Windows Identity Foundation responsible for reading the raw, serialized Security Token XML and converting it into a SecurityToken object. In the simplest case (such as in SharePoint), a single collection of SecurityTokenHandlers is associated with a web application in its web.config file. Incoming tokens are passed to the configured handlers one by one in order, and the first one that can successfully parse the XML returns a valid SecurityToken object. If you review the snippet from the microsoft.identityModel section of SharePoint's web.config above you'll find that the three token handlers configured in SharePoint 2010 are (in order):

1. Microsoft.IdentityModel.Tokens.X509SecurityTokenHandler
2. Microsoft.SharePoint.IdentityModel.SPSaml11SecurityTokenHandler
3. Microsoft.SharePoint.IdentityModel.SPTokenCache

The types of tokens handled by the first two entries are self-evident. The X509 handler is used to parse the signing certificate of incoming SAML tokens and the SAML1.1 handler parses the actual SAML tokens themselves. The SPTokenCache handler is a special handler for managing session security tokens (aka Security Context Tokens, or SCTs). We'll get to those when we get to the Session Authentication Module.

Once the SecurityToken is created from the RSTR, it's used to create a Claims principal. This principal is passed through any custom ClaimsAuthenticationManager (described above) to leave us with the final Claims principal that will be set on the HTTP and thread context for ongoing use in the web application.

What's Next?

And we're done! Well, not quite… Didn't think it was that easy did you? If this were the end, you'd have to re-authenticate each time you requested a new page in the web application. Fortunately, we create a special Session Security Token that can be used to maintain a single authenticated secure conversation across multiple messages. This is the responsibility of the Session Authentication Module, which we'll leave for another article.

Conclusion

Hopefully you now understand how WIF implements the passive requestor pipeline defined in the WS-Federation spec. In the next article, we'll cover the modifications SharePoint makes to this process.

01 - SharePoint, Claims, WS-* and the Windows Identity Foundation (WIF)
02 - WS-Federation Authentication Module and SharePoint Extensions
03 - The SharePoint Federation Authentication Module
04 - Integrating SharePoint with Windows Azure Active Directory (ACS)
05 - Session Authentication Module and WS-SecureConversation
06 - SharePoint Session Authentication Module
07 - Windows Claims Authentication
08 - Service (ActAs) Tokens

Thank you to Steve Peschka for reviewing these articles!

SharePoint Claims Authentication Part 1

SharePoint 2010 introduced Claims authentication based on the WS-Trust, WS-Federation, and SAML standards, bringing new opportunities for federation, authentication, and authorization scenarios. It also brought a brave new world of concepts and patterns for SharePoint administrators and developers to learn and master. In this series of articles, I'll take you on a deep dive through the mechanisms of the Windows Identity Foundation (WIF) Claims authentication pipeline with a special focus on its application to SharePoint. By understanding these from the inside out I hope you'll be better prepared to implement, maintain, extend and troubleshoot SharePoint, Azure, and O365 integration and federation and maximize your organization's investments in these and other products. As we progress I hope you'll not only find yourself better understanding Claims authentication within SharePoint, but also better understanding Claims authentication in general.

Why Claims?

You may be forgiven at this point for asking why. Kerberos has been a relatively stable protocol for many years and is still supported in SharePoint and other Windows applications. NTLM, warts and all, continues to function effectively in many environments. Nevertheless, several factors should drive you to seriously consider Claims authentication.

First, Kerberos and NTLM haven't been significantly updated in many years. Claims and complementary security protocols such as OAuth, OpenID and SAML offer new ways to delegate authentication, authorization, and user profile and attribute information in ways not possible with only the older protocols. Second, NTLM and Kerberos in Windows are not particularly well-suited for interoperation with other applications and systems. Within a single organization Windows authentication alone may be effective; but when federating with other organizations and working across the Internet, SAML and other Claims-based forms of authentication are a more common and supported option. Even within organizations, Claims significantly improves interoperability and coordination between Windows applications such as SharePoint and non-Windows applications and systems. For example, SharePoint's Business Connectivity Services leverage the interoperability made possible by Claims authentication to allow data from external LOB applications to be integrated with SharePoint.

Finally, Claims Authentication is not only the wave of the future, it's already well-established in the present. Microsoft and non-Microsoft systems and services alike utilize and offer Claims-based authentication already. Windows Azure Active Directory (aka Azure Access Control Service) and Active Directory Federation Services (ADFS) are based on Claims authentication. In .NET 4.5, the Claims Principal will become the base identity for all threads (as detailed in Vittorio Bertocci's blog post here). On the Internet, when you sign in to Windows Live ID or grant access to your Facebook profile you are utilizing WS-Federation and OAuth. To allow your applications, SharePoint included, to interoperate with all these other systems and services, using and understanding Claims authentication is a necessity.

The Claims Authentication Stack

Claims authentication is a process involving many participants and components. While many blog posts and technical articles are available to help you set up Claims authentication in SharePoint and other applications, my goal in this series is to take you on a step-by-step journey through all the components and protocols involved. If I miss something or you disagree with the way I describe things let me know in the comments and I'll respond or amend the content as appropriate. Occasionally I may leave out a detail or two to keep us focused on the fundamentals, but as you'll discover my hope is to share as much as possible. We'll focus primarily on the Claims authentication stack provided by Windows Identity Foundation (WIF), especially as it is extended and applied in SharePoint.

The foundation of the Claims authentication process in WIF and other authentication stacks is a number of standards and specifications managed by the Organization for the Advancement of Structured Information Standards (OASIS). These include the following:

WS-Security: Security Tokens

All of these standards are based on the concept of a security token, defined in WS-Security as follows:

A security token represents a collection (one or more) of claims. (WSS: SOAP Message Security, Page 11, Line 326)

Well-known types of security tokens include X.509 certificates, Kerberos tickets, SAML tokens, and, yes, Username/Password tokens. The claims asserted in a security token can include subjects, attributes (e.g. email addresses, names, and identifiers), symmetric and asymmetric (public) keys, passwords, and more. These claims often form the basis for authentication and authorization decisions, and may also be utilized to populate profiles or sign other tokens and messages.

Let's take an X.509 certificate token as an example. It specifies claims including the name of an entity and a public key associated with that entity. If a receiver of the X.509 token trusts this information, it can utilize the public key to verify that other messages signed by the associated private key truly originated from the associated entity. In addition, X.509 certificates are signed by another security token; this signature represents another claim asserted by the signing X.509 certificate that the entity in the signed certificate is the proper owner of the included public key.

To be useful, tokens received from a third party are almost always signed. The WS-Security spec defines a signed token as follows:

A signed security token is a security token that is asserted and cryptographically signed by a specific authority (e.g. an X.509 certificate or a Kerberos ticket). (WSS: SOAP Message Security, Page 12, Line 330)

However, just because a token has been signed by its issuer doesn't mean that the party presenting it to some other application is the proper owner and user of the token. For example, it could have been stolen in transit. To help confirm that the bearer of a token is in fact the proper owner of that token, some tokens, such as X.509 certificates and Kerberos tickets, associate the owner with a security key known only to that owner (e.g. a private key). This allows receivers to guarantee that messages signed by the specified private key originate from the proper owner of the token by checking that these messages have been signed by the associated key.

On the other hands, some tokens, notably SAML bearer tokens, don't rely on the bearer possessing a key; proof of ownership is demonstrated simply through possession. This is similar to cookie-based authentication, where the possessor of the cookie is assumed to be its rightful owner. Of course, HTTPS or some other transport-level encryption is essential in guaranteeing that the token or cookie hasn't been stolen. The concept of a bearer token comes up a lot in current Claims authentication, so come back and reread these last couple paragraphs in the future if you need to.

In the Windows Identity Foundation, Security Tokens are created, read, and validated by SecurityTokenHandlers. Pre-programmed handlers available out-of-the-box include a Kerberos handler, a SAML1.1 handler, a SAML2 handler, and a X509 handler, among others. SharePoint only supports handling SAML1.1 tokens directly.

WS-Trust: Requesting and Receiving Security Tokens

Having defined a security token, we now proceed to define how one goes about acquiring one. In some cases, such as X.509 certificates and Kerberos tickets, the token is acquired via "legacy" protocols such as via a Kerberos Ticket Granting Service or a X.509 Certificate Authority. For newer token formats such as SAML tokens, WS-Trust provides a protocol for requesting and receiving a token. In WS-Trust, the requestor sends a RequestSecurityToken (RST) request to a Security Token Service (STS) asking that STS to issue a security token asserting specified claims. Once the STS has satisfied itself that the requestor is entitled to the requested claims, it responds with a RequestSecurityTokenResponse (RSTR) containing the requested security token and any associated keys needed to use it.

For example, an entity with a Windows identity may request a SAML token specifying their UPN and information about AD groups they are part of. The requesting entity would craft an RST specifying the desired claims. It would then create a SOAP message including the RST in the body and a Kerberos ticket token in the header (to prove its rights to these claims) and submit this message to an STS. The STS would verify the Kerberos token and then create a SAML token with the requested claims. It would place the SAML token in an RSTR and place the RSTR in a SOAP message, which it would then send back to the requesting entity. The requesting entity would extract the new SAML security token from the message and use it as needed.

The Security Token Service provided with WIF and the STS implemented by SharePoint as the Security Token Service Application are both WS-Trust STS's. That is, they expect and process RSTs submitted directly in SOAP-based Issue requests, and send RSTRs back to the requestors in SOAP messages. (Note: the WIF STS can also act as a WS-Federation endpoint; more on that soon.)

Another common protocol for requesting and receiving security tokens is SAMLP (SAML Protocol). Although SAMLP is part of the SAML2 specification, you should be careful not to confuse SAML tokens with the SAML protocol. SAML tokens are XML security tokens asserting a collection of claims. The SAML protocol (SAMLP) is a protocol for requesting SAML (or other) security tokens. Currently, WIF and SharePoint do not support SAMLP, although ADFS does.

By understanding WS-Security and WS-Trust we've come a long way, but one element remains to be explained. The WS-Trust protocol just described works well in scenarios where an entity knows how to actively request and process a security token. This is fine for service-to-service communication, where the requesting service can be programmed to do just that. Web browsers, however, do not know how to craft RSTs and process RSTRs and security tokens. So another layer is needed to define how to request and utilize security tokens in end-user web applications (e.g. SharePoint). This brings us to…

WS-Federation: Web Browsers (i.e. Passive Requestors)

WS-Federation, specifically Section 13, contains a profile describing how to utilize the WS-Trust request/response protocol via a web browser, also referred to as a "passive" requestor. The simplest form of the process is illustrated in the following image from the spec:

(Web Services Federation Language v1.2, page 95, Line 3173)

When the browser requests a resource (e.g. a web page) without the required token (1), the web application (also known as a Relying Party, or RP) redirects the browser to an STS to get the needed token. It does this by replying to the request with a 302 Redirect response specifying the address of the STS endpoint to which the client should connect (2). The web application appends details about the token needed by adding a query string to this redirect URL. The web browser follows the redirect and sends the URL with query parameters to the STS endpoint (3). The endpoint authenticates the end-user (without going into details, this could be via a Kerberos ticket or a username/password combination) (4) and then prepares a RSTR to send back to the original web page. Here again, the browser cannot handle the RSTR on its own. Instead, the STS needs to redirect the browser with the appropriate RSTR back to the original web page (5). A 302 Redirect isn't enough in this case, as 302s can only perform GET requests to the destination URL and an RSTR is often too big to fit in a query string. A POST request to the original RP is initiated by setting some script in the response at step 5 that posts the RSTR to the original web page via the browser (6).

If all goes well, at this point the original web application will have an RSTR from the STS. It (the web application) can extract the security token from the RSTR and use it to authenticate the user and send back the requested resource (7).

Following along with the passive requestor profile

To help clarify this, let's follow the process when I access the management page for a namespace of mine in Windows Azure Active Directory (Azure Access Control Service). The management page is at https://joshgav.accesscontrol.windows.net/v2/mgmt/web. When I browse to this address in Internet Explorer I receive the following response:

HTTP/1.1 302 Found
Location: https://joshgav.accesscontrol.windows.net/v2/wsfederation?wa=wsignin1.0&wtrealm=https%3a%2f%2fjoshgav.accesscontrol.windows.net%2fv2%2fmgmt%2fservice&wreply=https%3a%2f%2fjoshgav.accesscontrol.windows.net%3a443%2fv2%2fmgmt%2fweb%2fsignin&wctx=%2fv2%2fmgmt%2fweb

I've highlighted the query parameter keys to make it somewhat easier to identify them. These parameters are all defined in the WS-Federation spec. wa stands for the Action requested; wsignin1.0 and wsignout1.0 are the two most common actions and represent sign-in and sign-out requests respectively. wtrealm stands for Trusting Realm and designates the security realm which the requested security token is intended for. wreply specifies where the user should be sent back to with the received RSTR. Note that most of the time this reply address is the same or a child of the Trust Realm (wtrealm) (and often only or the other is specified). wctx is an optional parameter for specifying context to be retained and replayed back to the original page in the reply.

The URL I'm being redirected to (https://joshgav.accesscontrol.windows.net/v2/wsfederation) is the endpoint for the WS-Federation passive STS endpoint for my own ACS domain. ACS, WIF, and ADFS can all provide a WS-Federation STS endpoint. This STS is now responsible for authenticating me and ultimately sending me back to the URL and page I originally requested. It could authenticate me with a simple name and password or numerous other options which we won't get into right now. Once I'm authenticated, it sends me back a page with the requested RSTR embedded in a form along with some JavaScript directing my browser to post this form back to the original URL. The returned page looks like this:

<html>
  <head>
    <title>Working...</title>
  </head>
  <body>
    <form method="POST" name="hiddenform" action="https://joshgav.accesscontrol.windows.net/v2/mgmt/web/signin">
      <input type="hidden" name="wa" value="wsignin1.0" />
      <input type="hidden" name="wresult" value="HTML-encoded RSTR goes here (decoded below)" />
      <input type="hidden" name="wctx" value="/v2/mgmt/web" />
      <noscript><p>Script is disabled. Click Submit to continue.</p><input type="submit" value="Submit" /></noscript>
    </form>
    <script language="javascript">window.setTimeout('document.forms[0].submit()', 0);</script>
  </body>
</html>

The RSTR goes where I've indicated in bold. If you'd like to see what an RSTR looks like, I decoded and included it below. Note that ACS uses a Simple Web Token (SWT) security token and not a SAML token; we won't worry about that right now. I took the liberty of converting the SWT from Base64 and URL decoding it so you'd have the pleasure of seeing what it actually contains. This is a bearer token, by the way. If it was still valid (and I hadn't modified the HMAC), you could take it right out of this blog post and authenticate to my management service with it.

<t:RequestSecurityTokenResponse Context="/v2/mgmt/web" xmlns:t="http://schemas.xmlsoap.org/ws/2005/02/trust">
  <t:Lifetime>
    <wsu:Created xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd">2012-04-25T20:07:47.376Z</wsu:Created>
    <wsu:Expires xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd">2012-04-26T04:07:47.376Z</wsu:Expires>
  </t:Lifetime>
  <wsp:AppliesTo xmlns:wsp="http://schemas.xmlsoap.org/ws/2004/09/policy">
    <EndpointReference xmlns="http://www.w3.org/2005/08/addressing">
      <Address>https://joshgav.accesscontrol.windows.net/v2/mgmt/service</Address>
    </EndpointReference>
  </wsp:AppliesTo>
  <t:RequestedSecurityToken>
    <wsse:BinarySecurityToken
      wsu:Id="uuid:73344c18-47a2-40d7-9ea7-1b63a8eb182c"
      ValueType="http://schemas.xmlsoap.org/ws/2009/11/swt-token-profile-1.0"
      EncodingType="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-soap-message-security-1.0#Base64Binary"
      xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd"
      xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd" >
        http://schemas.microsoft.com/ws/2008/06/identity/claims/role=Administrator
        &http://schemas.microsoft.com/accesscontrolservice/2010/07/claims/identityprovider=uri:WindowsLiveID
        &Audience=https://joshgav.accesscontrol.windows.net/v2/mgmt/service
        &ExpiresOn=1336035077&Issuer=https://joshgav.accesscontrol.windows.net/
        &HMACSHA256=e3JilQ6AHWCH5FtSwqXxEEgQ6mUF6gOa1PvoyAi65fI=
      </wsse:BinarySecurityToken>
    </t:RequestedSecurityToken>
  <t:TokenType>http://schemas.xmlsoap.org/ws/2009/11/swt-token-profile-1.0</t:TokenType>
  <t:RequestType>http://schemas.xmlsoap.org/ws/2005/02/trust/Issue</t:RequestType>
  <t:KeyType>http://schemas.xmlsoap.org/ws/2005/05/identity/NoProofKey</t:KeyType>
</t:RequestSecurityTokenResponse>

Once this form with its keys and values is posted back to the original requested URL (the ACS management page), the web application extracts the presented token, checks my authorization for the page requested and (if I'm authorized) serves me said page. In the case of the ACS Management service and most web apps, a cookie is set at this point so that the token won't need to be requested and/or processed for each subsequent request.

Conclusion

At this point we've covered the WS-Security, WS-Trust, and WS-Federation protocols. We've touched on SAML and other token profiles such as Kerberos, X.509, and even SWT. There's still quite a long ways to go in explaining how WIF automates and coordinates all of these elements for you, but with these foundations you'll find the ensuing conversations easier to follow.

In this post, we'll discuss the information returned by SPList and SPListItem objects and a way to retrieve the actual columns and values stored in a SharePoint list as first-class properties of PowerShell objects. To assist with this you can utilize the SPList module attached to this post. Store the module folder in one of the paths in the $env:PSModulePath variable (e.g. \windows\system32\WindowsPowerShell\v1.0\Modules) and run the following on a SharePoint server to get started:

PS:> Import-Module SPList

To begin, let us discuss how to access information about lists via PowerShell. In the SharePoint hierarchy, the final level for which an out-of-the-box cmdlet exists is the SPWeb object. SPWeb objects represent individual sites in a site collection and contain lists and folders. The simplest way to retrieve all lists in a SharePoint site is as follows (of course, change the value for $SiteUrl to a URL in your farm):

PS:> $SiteUrl = "http://sharepoint10"
PS:> $web = Get-SPWeb $SiteUrl
PS:> $web.Lists | Sort-Object -Property Title | Format-Table -Property Title

No default formatting is defined for SPList objects, so if you don't specify a little formatting via the Format-Table command (as above) or a custom format.ps1xml file you'll end up with way too much information scrolling down your screen. The module included with this post includes a format.ps1xml file which causes SPList objects to be returned as a nicely formatted table or list by default.

If you like, the module available with this post includes a Get-SPList function. The function requires a list name and a SPWeb reference. The SPWeb reference can be an actual URL or can be taken from the pipeline. Two usage examples for the Get-SPList function follow:

PS:> $SiteUrl = "http://sharepoint10"
PS:> Get-SPList -ListName *Announce* -Web $SiteUrl
PS:> Get-SPSite -Identity $SiteUrl | Get-SPWeb | Get-SPList *announce*

The latter example will retrieve all lists with the letters "announce" in their title in all sites (webs) in the specified site collection. Try it!

Once you have a list you can retrieve the items available in that list. Doing so with the object model would look like this:

PS:> $ListName = "Announcements"
PS:> $List = $web.Lists[$ListName]
PS:> $List.Items | Format-Table -Property Title, ID

Again, by default the formatting isn't particularly pretty so we fix it up with Format-Table. And again, the module included with this post includes some prettier formatting instructions for SPListItem objects.

Compare the output of properties and values returned for a list item in PowerShell with the actual column names and values in the SharePoint list (for example in the Web UI). You'll discover that the information returned in PowerShell (from the object model) is actually only metadata about the list item, not the actual list item column names and values themselves. We'll shortly introduce a function which will retrieve the actual data for each SharePoint list item, but first let's see how you'd accomplish this via the object model itself.

First, you generally have to know the name of the column you're looking for in advance. We'll use the Body column in the Announcements list. To retrieve the values for the Title and Body fields for these list items, do this:

PS:> $List.Items | Format-Table -Property Title, @{Label="Body";Expression={$_["Body"]}}

Not pretty is it? Notice how we have to use index notation and hardcode the name of the column we want to retrieve. This is brittle - for example if the list item happened to not have a "Body" column we'd likely get an error. We also have little ability to pipe these results down the pipeline for further evaluation and filtering. For these reasons I've written a function called Get-SPListItem which returns an object whose members (properties) represent the actual column names in the SharePoint list and their respective values for each list item. It's part of the module included with this post and usage looks like this:

PS:> $WebUrl = "http://sharepoint10"
PS:> Get-SPList -ListName Announcements -Web $WebUrl | Get-SPListItem

That's it! The objects it returns contain a property BaseListItem which refers to the original list item (with object metadata) and then a variable number of dynamic properties which reflect the actual data stored on the list item. The way it works is by scanning the list of fields associated with each list item via that item's Fields property and then creating a dynamic property named after the field and with the value stored on the item for that field. Because it looks at the Fields property for each list item it works even on Lists that contain multiple content types (and hence different Fields collections for each list item).

If you need the metadata directly for a list item and want to use the function, I've included an "-OriginalOnly" switch which will return the original SPListItem object.

One of my original uses for these functions was to help in reviewing the "Review Problems and Solutions" list generated by the Health Analyzer after its checks. Items get added to this list and are assigned a HealthReportSeverity of 0-4, with 0 representing a critical error, 1 an error, 2 a warning, and 4 success. Using the functions in the SPList module you can automatically retrieve latest results and even filter by severity. Based on what you find, sending an email message or otherwise tripping some sort of alarm shouldn't be too difficult. Following are some genericized examples which should help you get started in any environment:

PS:> $CAWebApp = Get-SPWebApplication -IncludeCentralAdministration | ? {$_ -is [Microsoft.SharePoint.Administration.SPAdministrationWebApplication] }
PS:> $CAWeb = Get-SPSite -WebApplication $CAWebApp | Get-SPWeb | ? {$_.Title -eq "Central Administration"}
PS:> Get-SPList -Web $CAWeb -ListName "Review Problems and Solutions"
PS:> Get-SPList -Web $CAWeb -ListName "Review Problems and Solutions" | Get-SPListItem
PS:> Get-SPList -Web $CAWeb -ListName "Review Problems and Solutions" | Get-SPListItem | Format-List *Health*
PS:> Get-SPList -Web $CAWeb -ListName "Review Problems and Solutions" | Get-SPListItem | ? {$_.HealthReportSeverity -match "^[012]"} | Format-List *Health*

With this background you're close to being able to set properties on lists and list items as well. For guidance, check out this post by fellow PFE Kirk Evans.

SPList.zip