But what would it be like to develop in a SharePoint environment if you didn’t have database server access? This is a typical situation at large enterprises. The database team is a separate group, possibly located in another time zone on the other side of the world. This team closely guards their turf, not allowing anyone outside their team to logon their database servers. You panic because you can’t touch SQL Server!

CAVEAT: Remember, directly accessing SharePoint databases is NOT supported. The following procedures illustrated what could be done in an emergency when a DBA is not available. This information is presented as a troubleshooting alternative in development and test environments only. These procedures should NOT be performed against a production farm.

But wait, you are not totally out of luck. First, let’s analyze the situation. What do you know?

1. You probably have an installation account (local server administrator) and password.
2. Similarly, you probably have the SharePoint farm account and password.
3. You know the farm account can connect to the SQL Server, after all, that is how the configuration database is accessed.

With this information, you probably have a door to the SQL Server. It may not be as easy as the point-and-click SQL Server Management Studio UI, but it is all you need. So what do you do?

1. Download the SQLCMD command line utility. This is a small (12 MB) msi package you can install on one of the SharePoint servers. Scroll down to the heading Microsoft SQL Server 2008 R2 Command Line Utilities. Quoting from that page:
   The SQLCMD utility allows users to connect to, send Transact-SQL batches from, and output rowset information from SQL Server 7.0, SQL Server 2000, SQL Server 2005, and SQL Server 2008 instances.
   Click the link X64 Package (SqlCmdLnUtils.msi) to download. If you are downloading to your local machine because of proxy or firewall restrictions on the SharePoint server, map a drive to the SharePoint server and copy the installation package to the SharePoint server.
2. Logon the SharePoint server as the installation account. Install the package, taking all the default options. The installation only takes a few seconds.
   
3. Optionally, update the PATH environment variable to make it easier to use SQLCMD in command prompt. Open the Control Panel > System > Advanced system settings. Click the Advanced tab, and then the Environment Variables button. In the bottom pane, System variables, scroll down to the Path variable and then click the Edit button.
   Go to the end of the path, and then add a semicolon and C:\Program Files\Microsoft SQL Server\100\Tools\Binn or whatever path where you installed SQLCMD. Click OK to close all the dialog windows.
   
4. Logoff off as the installation account and logon as the farm administrator account so you are running in the security context of the farm account, or alternatively, open a command prompt and Run as the farm account.
5. If you don’t know the SQL Server instance name from memory, open Central Administration > System Settings > Manage servers in this farm, and find the database server under Farm Information at the top of the page.
6. Open a command prompt and begin entering Transact-SQL commands.

The farm administrator has securityadmin and dbcreator roles already (else the farm wouldn’t work), so you can submit T-SQL commands.

First, connect to the server by entering:

sqlcmd –S <database-server-instance-name>

To find what accounts have server roles, enter:

EXEC sp_helpsrvrolemember
GO

To display all the database names, enter:

SELECT SUBSTRING(name,1, 64), database_id FROM sys.databases ORDER BY name
GO

To look at information for a specific database, you must switch context to that database with the USE command. The following switches context to the Secure Store Service database.

USE SecureStore2_DVPROD_shared
GO

To find the DBOs for the database, enter:

EXEC sp_helprolemember 'db_owner'
GO

To get the database sizes, create a SQL batch file with notepad. Enter the following SQL statements and save the file as DatabaseFileSizes.sql.

SELECT
    databasename = DB_NAME(dbid),
    name,
    fileid,
    drive = LEFT(filename,1),
    filename,
    filegroup = FILEGROUP_NAME(groupid),
    'size KB' = CONVERT(nvarchar(15), CONVERT(bigint, size) * 8),
    'maxsize KB' = (CASE maxsize WHEN-1 THEN
              N'Unlimited'
          ELSE
              CONVERT(nvarchar(15), CONVERT(bigint, maxsize) * 8) END),
    'growth' = (CASE status & 0x100000 WHEN 0x100000 THEN
              CONVERT(nvarchar(15), growth) + N'%'
          ELSE
              CONVERT(nvarchar(15), CONVERT(bigint, growth) * 8) END),
    'usage' = (CASE status & 0x40 WHEN 0x40 THEN 'log only' ELSE 'data only' END)
FROM sys.sysaltfiles
ORDER BY drive, size

Next, exit the current SQLCMD session by entering EXIT.

Enter the following command which references the file just created batch file as the input file:

sqlcmd –S <database-server-instance-name> –I .\DatabaseFileSizes.sql –O .\DatabaseFileSizes.txt

The output file (DatabaseFileSizes.txt) can be imported into Excel for further analysis; such as, tracking trends from month-to-month.

These examples should give you a good taste of what is possible. You are limited only by the farm account permissions and your knowledge of T-SQL.

The Central Administration interface only provides a means to schedule incremental imports.

A full import can be started through Central Administration on-demand by using the Start Profile Synchronization link, but this link does not provide for scheduling.

It is possible to schedule full imports using the Windows Server Task Scheduler and a Windows PowerShell script.

1. Create the Windows PowerShell script ExecuteFullImport.ps1 and associated configuration file ExecuteFullImport.xml. Place in the standard scripts folder of the server.   (Source code below. Special credit goes to Brian Lalancette and the AutoSPInstaller project for foundational code included in this script, reference https://autospinstaller.codeplex.com. AutoSPInstaller is a fantastic script to build multi-server SharePoint 2010 farms.)
   
2. Create a schedule task. Start Menu > Administrative Tools > Task Scheduler;
   or Run > taskschd.msc.
   
3. Right click the Task Scheduler Library node in the left pane, and select Create Task.
   
   
4. On the General Tab, enter the Name and Description.
   
   
5. On the Triggers Tab, click the New button, and then enter the desired schedule.
   
   
6. On the Actions Tab, select Start a program, enter PowerShell as the program/script, enter the script name ExecuteFullImport.ps1, and Start in put the directory containing the script and associated xml configuration file.
   
   
7. Optionally, modify the Conditions and Settings tabs’ configuration.
   
8. The script will run at the scheduled time. The PowerShell window remains open by default. Press any key to close the window.
   

ExecuteFullImport.ps1

############################################################
### Powershell commands to execute a User Profile full import
############################################################

param
(
    [string]$InputFile = "ExecuteFullImport.xml"
)

###############################################################
### ExecutionPloicy Note: errors/warnings expected
###############################################################

Set-ExecutionPolicy -ExecutionPolicy Unrestricted -Force -confirm:$false
write-host "Displaying execution policy" -foregroundcolor yellow
Get-ExecutionPolicy -List

$0 = $myInvocation.MyCommand.Definition
$dp0 = [System.IO.Path]::GetDirectoryName($0)

# Get the configuration file contents
$configPath = Join-Path -Path $dp0 -ChildPath $InputFile
[xml]$xmlinput = (Get-Content $configPath)

#region functions
###############################################################
# Note: Following functions borrowed from AutoSPInstallerFunctions.ps1
Function StartTracing
{
    $LogTime = Get-Date -Format yyyy-MM-dd_h-mm
    $script:LogFile = "$env:USERPROFILE\Desktop\ExecuteFullImport-$LogTime.rtf"
    Start-Transcript -Path $LogFile -Force
}
 
Function WriteLine
{
    Write-Host -ForegroundColor White "--------------------------------------------------------------"
}

Function Pause
{
    #From https://www.microsoft.com/technet/scriptcenter/resources/pstips/jan08/pstip0118.mspx
    Write-Host "Press any key to exit..."
    $null = $host.UI.RawUI.ReadKey("NoEcho,IncludeKeyDown")
}
###############################################################
#endregion functions

StartTracing
If (!$env:StartDate) {$env:StartDate = Get-Date}
Write-Host -ForegroundColor White "-----------------------------------"
Write-Host -ForegroundColor White "| ExecuteFullImport script        |"
Write-Host -ForegroundColor White "| Started on: $env:StartDate |"
Write-Host -ForegroundColor White "-----------------------------------"

Try
{
    # The SharePoint PowerShell snap-in is required
    If ((Get-PSSnapin |?{$_.Name -eq "Microsoft.SharePoint.PowerShell"})-eq $null)
    {
        WriteLine
        Write-Host -ForegroundColor White " - Loading SharePoint Powershell Snapin"
        Add-PsSnapin Microsoft.SharePoint.PowerShell -ErrorAction Stop
        WriteLine
    }
   
    $stopTimerJob = $xmlinput.Configuration.Farm.FullImport.StopCleanUpTimerJob
   
    if ($stopTimerJob)
    {
        $timerJob = Get-SPTimerJob "mysitecleanup"
        Disable-SPTimerJob $timerJob
   
        WriteLine
                    Write-Host -ForegroundColor White " - IMPORTANT: My Site Clean Up time job has been disabled."
                    Write-Host -ForegroundColor White " - Remember to enable the timer job after the import completes."
        WriteLine
    }

    # Get the Profile Service Application
    $upa = (Get-SPServiceApplication | ? {$_.GetType().ToString() -eq "Microsoft.Office.Server.Administration.UserProfileApplication"})
   
    # Start the full profile import job
    # NOTE: This method call appears to "eat" exceptions, so it "fails silently". Check the application event log for error details.
    $upa.StartImport($true)
   
    Write-Host -ForegroundColor White " - Full Import request submitted, monitor progress with the Central Administration User Profile Service Application page"

}
Catch
{
    WriteLine
    Write-Host -ForegroundColor Yellow " - Script aborted!"    
    If ($_.FullyQualifiedErrorId -ne $null -and $_.FullyQualifiedErrorId.StartsWith(" - "))
    {
        # Error messages starting with " - " are thrown directly from this script
        Write-Host -ForegroundColor Red $_.FullyQualifiedErrorId
    }
    # Lately, loading the snapin throws an error: "System.TypeInitializationException: The type initializer for 'Microsoft.SharePoint.Utilities.SPUtility' threw an exception. ---> System.IO.FileNotFoundException:"...
    ElseIf ($_.Exception.Message -like "*Microsoft.SharePoint.Utilities.SPUtility*")
    {
        Write-Host -ForegroundColor Yellow " - A known (annoying) issue occurred loading the SharePoint Powershell snapin."
        Write-Host -ForegroundColor Yellow " - We need to re-launch the script to clear this condition."
        $ScriptCommandLine = $($MyInvocation.Line)
        Write-Host -ForegroundColor White " - Re-Launching:"
        Write-Host -ForegroundColor White " - $ScriptCommandLine"
        Start-Process -WorkingDirectory $PSHOME -FilePath "powershell.exe" -ArgumentList "$ScriptCommandLine" -Verb RunAs
        Start-Sleep 10
        Exit
    }
    Else
    {
        #Other error messages are exceptions. Can't find a way to make this Red
        $_ | Format-List -Force
    }

    $env:EndDate = Get-Date
    Write-Host -ForegroundColor White "-----------------------------------"
    Write-Host -ForegroundColor White "| ExecuteFullImport script        |"
    Write-Host -ForegroundColor White "| Started on: $env:StartDate |"
    Write-Host -ForegroundColor White "| Aborted:    $env:EndDate |"
    Write-Host -ForegroundColor White "-----------------------------------"
}
Finally
{
    Stop-Transcript
    If ($ScriptCommandLine) {Exit}
    Else {Pause}
    Invoke-Item $LogFile
}

<?xml version="1.0" ?>
<Configuration>
  <Farm>
    <FullImport
            StopCleanUpTimerJob="true"
            />
  </Farm>
</Configuration>

OK, so start checking Kerberos configuration:

1. SPNs created correctly for the service accounts and web applications? Check!
2. Users successfully authenticated by Kerberos when directly added to the SharePoint group? Check!
3. The domain local group is in the same domain as the SharePoint farm? Check!
4. Can the user access a new web application in the SharePoint farm using NTLM and the local domain group? Check!

Everything checks out, so why is access denied when the domain local group is used for a web application configured with Kerberos authentication????

OK, let’s take a step back and think about this. What is the definition of a domain local group? Looking at https://technet.microsoft.com/en-us/library/dd861330.aspx we find:

Domain local groups

Members of domain local groups can include other groups and accounts from Windows NT, Windows 2000, Windows Server 2003, Windows Server 2008, and Windows Server 2008 R2 domains. Members of these groups can be assigned permissions only within a domain.

Groups with domain local scope help you define and manage access to resources within a single domain. These groups can have the following as their members:

• Accounts from any domain
• Global groups from any domain
• Universal groups from any domain
• Domain local groups, but only from the same domain as the parent domain local group
• A mixture of any of the above

Our domain local group complies with this definition, the members are domain accounts, the SharePoint farm is in the same domain, etc.; so again, why is access denied using Kerberos but not NTLM? It’s time for some deeper investigation.

SharePoint 2010 in Windows Classic mode keeps an internal token with the security identifiers (SIDs) of the user, and all the groups the user is a member of. This token is matched against the secured object’s ACL to determine if the user has access either by the user SID directly, or by membership in a group. This seems to imply our user’s token must be missing the domain local groups SID – but why?

How does SharePoint populate the list of SIDs in the token? SharePoint pulls the SID list from the Kerberos ticket. Looking at https://blogs.msdn.com/b/openspecification/archive/2009/04/24/understanding-microsoft-kerberos-pac-validation.aspx we find this high level description of the Kerberos ticket contents.

The Privilege Account Certificate (PAC) is an extension element of the authorization-data field contained in the client's Kerberos ticket. The PAC structure is defined in [MS-PAC] and conveys authorization data provided by domain controllers (DCs) in an Active Directory-enabled domain. It contains information such as security identifiers, group membership, user profile information, and password credentials. The illustration below shows the relationship between a Kerberos ticket and PAC.

Noting the phrase conveys authorization data provided by domain controllers (DCs) , what DC is being used to generate the Kerberos ticket? To answer that, think through the ticket granting steps:

1. What identity is used to request a ticket from the DC (remember the DC is also the KDC)? The SPN account.
2. What is the SPN account for a web application? The web app’s application pool identity.
3. What is the application pool identity? admin\AppPool1.
4. What DC is contacted? The first DC in the admin.contoso.com domain that responds to the request.

And now one final question:

1. Which domain contains the domain local group and SharePoint farm? prod.contoso.com.

The light bulb comes on! The service account (admin\AppPool1) is in a different domain from the users and SharePoint farm. The DC in the admin domain that generates the Kerberos ticket has no knowledge of domain local groups in the prod domain, since by definition, domain local groups are scoped to the local domain. The Kerberos ticket presented to SharePoint lacks the SIDs of the prod domain local groups, hence access is denied.

How can this be resolved? That depends upon the existing investment in domain local groups and the organization’s security policies. Some possibilities are:

1. Promote the domain local groups to universal groups. This may be painful if there are hundreds or thousands of domain local groups. Besides the logistics of group promotion, you have to consider the additional DC replication traffic, and the possibility of breaking other legacy applications depending upon domain local groups.
2. Move the SPN service accounts to the same domain as the SharePoint farm. This allows continued use of domain local groups, but may violate the organization’s security model;  that is, the admin domain may be used for service accounts due to other security considerations. This approach has side effects, like the need to get exceptions for the SharePoint service accounts, the need to delete and recreate the SPNs, and the need to modify GPOs that control password changes.

PS:

Another interesting factor to consider. Testing access to a site needs to be done using the user account itself. The Effective Permissions feature in Site Settings does not acurately reflect permissions granted by group membership unless the user has actually visited the site since the last group membership change.

There is a TechNet article that talks about this, https://technet.microsoft.com/en-us/library/ff431687.aspx#bkmk_ins_exis_farm. This rather long article includes this sentence: Start the services on the Web front-end servers on which you want users to view and edit documents in the browser. This brief guidance is understandable, since it probably covers many simple deployments, and a more complete answer lengthen the article and make it harder to consume, but for larger farms, this simple guidance may not be optimal.

First, let’s establish some background on service instances and service applications in general. Service instances can run on any, or all, servers in a farm; both WFE and APP servers in any combination. The magic that allows this is the application topology service. The topology services makes service consumers (the WFE service proxies in this case) aware of the servers where the service instances are running. The service instances then load balance themselves through their built-in load balancer and proxies to spread the load evenly. The service consumer and the service instances DO NOT have to be on the same server. A common example is the managed metadata service, which might only have instances running on 2 APP servers. Requests for term sets are relayed from the WFEs to the appropriate APP server via the service proxies and load balancers. The key point is the service instance does not have to run on the WFE.

In the case of Office Web Apps, it is not necessary for the OWA services to run on the WFEs, although that may make sense in simple scenarios. For example, there could be just 2 WFEs running only the web application service, while there are 8 APP servers running the OWA service instances with the built-in load balancing spreading the load from the 2 WFEs across the 8 APP servers. (You will see this taken even further in SharePoint 2013 where OWA services run in an entirely separate farm from SharePoint)

Taking one example, what if Excel Services is rarely used? It does not make sense to run instances of Excel Services on all the WFEs? This just takes up WFE memory and CPU that might better be used to respond to normal page requests.

Taking another example, the other extreme is all users heavily hit Excel Services. In this case the WFEs may not be able to handle the Excel Service load, so it would be better to stand up APP servers dedicated to Excel Services, and scale out by adding additional APP servers so the WFEs are not bogged down doing Excel calculations.

So the long answer of where to put the OWA service instances is “it all depends”. Analyze the expected usage of each service (Word Viewer, PowerPoint Viewer, and Excel Services), and then start the necessary number of instances on the most appropriate servers.

When you first open General Settings dialog for a new web application, it shows the selected quota template as “No Quota”. As we will see below, this is misleading.

We created the My Site hosting web application using a PowerShell script. That script also created a default quota template per the customer maximum and warning size requirements, and then updated the web application’s default template to that value. We named this quota template “MySite Quota”.

The problem is SharePoint doesn’t use the web application default template when creating personal sites. A little digging shows that SharePoint creates its own default personal site quota template using the hard coded name “personal site” (or the equivalent installed language translation). When creating a personal site, this template is applied, not the web application default template.

I suspect SharePoint creates the default template for these reasons:

1. To make sure the template is available; that is, to avoid configuration errors if the customer forgets to create a quota template.
2. To specify resource usage limits for Sandbox solutions. (I bet many customers don’t realize Sandbox solutions can be deployed to personal sites.)

If you later need to override the default quota for certain users, you can go to Central Administration > Site Collection Quotas and Locks (or use PowerShell). You can change the template to another template you have created (like “gold personal site” in the following picture), or change the template to “individual quota” and manually enter values in the following fields.

More information regarding the personal site quote template can be found at Plan for My Sites.

Built-in load balancing simplifies deployment and administration, but has user impacts that might not be obvious. We discovered some of these impacts recently. A customer asked us to test Service Application load balancing in their environment. We had to test both shared services (services published from a publishing farm to a trusted consuming farm); as well as, local services.

Discovery 1—Service Application Status Doesn’t Change

The Status column in Central Administration > Manage Service Applications doesn’t change when all instances of a service application are stopped. We tested stopping all instances of Managed Metadata, Business Connectivity, and PowerPoint. In each case, Central Administration continued to show the status as Started even though no service instances were running.

Discovery 2—Consuming Farm May Not Know of Failures for Up to 15 Minutes

The blog referenced above, What is the Service Application Load Balancer, explains that a publishing farm checks availability of a published service once every 15 minutes by running the Application Addresses Refresh job. This allows a 15 minute window during which the users in the consuming farm might see errors; for example, with the Managed Metadata Service management page. This might be confusing to the farm administrator, because Discovery 1 shows that Central Administration reports the status is Started.

What is more insidious is that other user facing pages depend upon the availability of the Managed Metadata Service to render correctly. An example is the User Profile update page. In this screen capture we can see the user is unable to update Ask Me About, with no direct mention of a Managed Metadata Service problem in the error text; when in fact, unavailability of the Managed Metadata Service was the root cause of this error in our testing.

A point to take away from this is if a service in a publishing farm fails; that is, all service instances have stopped, after starting one or more instances immediately, in the publishing farm go to Central Administration > Monitoring > Review Timer Job Definitions, click on Application Addresses Refresh job, and click the Run Now button to make consuming farms aware of the newly available service end points.

Discovery 3—Excel Calculation Service May Not Use Available Service Instances

So far as I know, Excel Calculation Service is the only service application with multiple load balancing algorithms. The load balancing choices are:

• Workbook URL
• Round Robin with Health Check
• Local (only is the service is running on the WFE)

The default is Workbook URL. This default has an interesting implication. Our testing showed a browser with a spreadsheet open prior to the service instance failure, continues to try to use the failed instance even though functioning service instances are available on other application servers. Apparently the Workbook URL creates a “sticky session” with the non-functioning application server. Users with the workbook already open get errors, but subsequent users of the same workbook are connected to a functioning service instance.

This raises one of those consulting questions of what is the best load balancing algorithm. It all depends. Workbook URL load balancing improves caching, responsiveness, and resource usage; while at the same time, for this admittedly rare use case, causing an unintentional single point of failure.

Discover 4—Cached Word and PowerPoint Documents Can Mask Service Failure

While this discovery is not a problem itself, it can mask a service application failure.

In our testing, we “viewed in browser” several PowerPoint files. As you may know, Office Web Application caches rendered documents to improve performance on subsequent requests. These documents continued to rendered in the browser even after all service instances were stopped; however, uncached documents returned an error.

This inconsistent behavior of some documents rendering and some not could lead to end user confusion, and even farm administrator confusion, until the root cause (all service instances stopped) is identified.

Discover 5—User Profile Service Has Various Impacts

As most administrators know, the user profile service application allows subscribing farms to make direct SQL calls to the user profile database in the publishing farm. As a consequence, only parts of the profile service application are dependent upon the functioning of service instances in the publishing farm. The user experience is therefore inconsistent in that some features continue to work, some features fail silently, and some features displays errors. Here is a quick list of what we saw when going to a My Site of a user who had previously created their My Site, added colleagues, and had site memberships.

1. My Site home page renders (https://my.contoso.com/default.aspx), with these consequences:
• My Colleagues list appears, but clicking a colleague name results in an error
• My Interests and Newsfeed Settings also result in errors
• Organization Browser fails to render, but doesn’t display an error either; that is, silently fails
2. My Content displays the personal site home page (https://my.contoso.com/personal/user-name/default.aspx)
3. My Profile (https://my.contoso.com/person.aspx) results in an error
4. The enterprise search center page (https://search.contoso.com/pages/default.asp) results in an error
5. A team site page (https://teams.contoso.com/default.aspx) renders, but any attempt to click on a user name results in an error

While scripting the deployment of a multiple farm SharePoint 2010 environment with a Shared Services farm providing services to consuming farms, the question arose, is it possible to publish a service application with PowerShell? Without this scripted capability, many manual steps would be necessary to build the farm. There are no published PowerShell cmdlets to publish service applications, so investigation was needed.

Using the Get-Member cmdlet to display all the properties of a service application object led to the discovery of these public properties and methods:

Armed with this knowledge, a service application can be published with the following PowerShell snippet:

$serviceApp = Get-SPServiceApplication | ? {$_.GetType().ToString() -eq "Microsoft.Office.Server.Administration.UserProfileApplication"}
$termsOfServiceUri = $null
$name = $serviceApp.name

Write-Host -ForegroundColor White " - Publishing Service Application $name ..."

$serviceApp.Shared   = $true
$serviceApp.Comments = "Shared Services Farm User Profile Application"

if (($termsOfServiceUri -ne $null) -and ($termsOfServiceUri.length -gt 0))
{
    $serviceApp.TermsOfServiceUri = $termsOfServiceUri
}

$serviceApp.Update()

The stretch goal then becomes, is it possible to grant a consuming farm permissions to the a published service application with PowerShell?

Again, with some research, the answer is yes – assuming you know the consuming farm ID. This PowerShell snippet grants the consuming farm access.

$ServiceAppSecurity = Get-SPServiceApplicationSecurity $serviceApp.Id
$claimProvider = (Get-SPClaimProvider | ?{$_.DisplayName -eq "System"})
$principal =  New-SPClaimsPrincipal "<CONSUMING-FARM-GUID>" -ClaimType "https://schemas.microsoft.com/sharepoint/2009/08/claims/farmid" -ClaimProvider $claimProvider.ClaimProvider

if ($principal -ne $null)
{
       Grant-SPObjectSecurity $ServiceAppSecurity -Principal $principal -Rights "Full Control"
   Set-SPServiceApplicationSecurity $serviceApp.Id -objectSecurity $ServiceAppSecurity -ErrorAction SilentlyContinue

   $value = $principal.Value
   Write-Host -ForegroundColor White " - $value granted Full Control Permission to Service Application $name."
}

We recently ran into an Access Denied problem using the Secure Store Service. The environment has a Shared Services Farm which publishes the Secure Store Service (along with BCS, Managed Metadata, User Profile, FAST Query SSA, and Web Analytics) to consuming farms (https://technet.microsoft.com/en-us/library/ff621100.aspx). Each farm has unique Farm Administrator accounts; as well as, unique service accounts for application pools and service applications.

After publishing the services, we set up a trust relationship between farms by exchanging certificates; and then in the consuming farm, we connected to the published shared services (https://technet.microsoft.com/en-us/library/ee704552.aspx). The consuming Farm Administrator account was given Full Control permission to all shared services, including the Secure Store Service. So far everything was working as planned. Search worked, user profiles and My Sites worked, we were feeling good.

Trouble occurred when the consuming farm administrator tried to create the unattended accounts for the local Excel and Visio service applications. As you know, unattended accounts required you to create Target Applications in the Secure Store Service. Creating a Target Application ID walks you through a wizard of 3 pages. In one case, every other page submission resulted in an Access Denied error. In the other case, we couldn’t get past the first page due to access denied. A look at the ULS logs of both the consuming and publishing farms didn’t reveal any clues, other than an access denied exception was being thrown in the web service call from the consuming farm to the topology service of the publishing farm. We kept asking ourselves, why would an account with Full Control permission to the service application get access denied? We even tried to create the Target Applications directly in the publishing farm logged in as the consuming farm account, thereby eliminating inter-farm trust as a factor. Still got access denied.

Searches of TechNet, the web, and Microsoft’s KBs didn’t provide any solution. We tried recreating the service application, recycling application pools, resetting IIS, even rebooting servers. The access denied error persisted. We must be missing something, but what?

We started looking at every aspect of the Secure Store Service configuration on the publishing farm, and then we found it! Even though there is a Full Control permissions displayed when you click the Permissions tool bar button, this permission apparently doesn’t really give full control. We discovered by clicking the Administrators tool bar button there are granular administrator rights for Create, Delete, and Manage Target Application. I can’t explain why these rights are administrative rights not covered by the Full Control permission; however, granting the consuming farm account these administrator rights solved the problem! Any other accounts needing these rights will also have to be added.

The step-by-step solution follows.

1. Open a browser and navigate to the Publishing Farm Central Administration web site using the publishing Farm Administrator account.
2. Navigate to Manage Service Applications.
3. Select the line containing Secure Store Service by clicking in a blank area. DO NOT click on the name itself.
   
4. Click the Administrators button in the task bar.
5. Enter the consuming Farm Administrator account (SPFarm_Q in this example) in the top text box, and then click the Add button.
6. Check the Create, Delete, and Maintain Target Application check boxes as shown in the screen shot.
   
   
   
7. Click the OK button.

Confusion often surrounds what privileges must be granted to developers to build and deploy solutions on a development server. It would seem the developer can be added to both the local machine administrator and farm administrator groups; and then be able to do anything. Unfortunately, the first time the developer tries to deploy a solution to a web application there are access denied errors. What’s going on here? Shouldn’t local administrator group and farm administrator group membership allow the developer unlimited access? To answer this question we need to have a high level picture of the security model.

Homework Assignment

First, I strongly encourage reading the TechNet Article, SharePoint 2010 Architectures Overview (https://technet.microsoft.com/en-us/library/gg552610.aspx). If you don’t have time to read the entire article, at least read the first 3 paragraphs of the section entitled Worker Processes, Farm Solutions, and Sandboxed Solutions; in fact, read it over-and-over until you have it memorized. This is time well spent.

Parallel Universes (The Twilight Zone)

Have you finished this homework assignment? OK, let’s jump into some of the security complexities.

Think of security as 3 parallel universes (remember the old Twilight Zone TV show?). A user in one universe may be blissfully unaware of the other universes; for example, a machine administrator can be denied access to the SharePoint universe and/or SQL universe, and vice versa.

• Universe One is the local server itself.
• Universe Two is SharePoint farm, logically hosted on the local server.
• Universe Three is the SQL Server, either a local or remote SQL instance. 

For a Universe One (local server) or Universe Three (SQL Server user) to cross into Universe Two (SharePoint farm), that user must be granted privileges within SharePoint; for example:

• Added to the farm administrators group through Central Administration
• Added to a web application user policy through Central Administration
• Added to as a service application administrator through Central Administration
• Added to a Site Collection group by a Site Collection administrator
• Granted database access with the Add-SPShellAdmin cmdlet.

For a Universe One (local server) or Universe Two (SharePoint farm) user to cross into Universe Three (SQL Server), that user must be granted a SQL Server login and made a database user for one or more databases. SQL access is normally granted as a side effect of granting a user access to the SharePoint farm. See the prior bullets for ways this can happen. If you feel you have to directly grant database access, then you probably have something else wrong in your security design.

For a Universe Two (SharePoint farm) user to cross into Universe One (local server), the user must be added to local machine groups. This normally occurs automatically as part of SharePoint installation, or as a side effect of other actions such as executing the Add-SPShellAdmin cmdlet which implicitly makes the named user a local administrator (be sure this is what you intended). If you feel you have to directly add SharePoint users to local machine groups, with few exceptions you probably have something else wrong in your security design. An example of an exception where manual group addition is required is to add developers to the Remote Desktop Users group so the developers can RDP onto the server without having to be local administrators.

These disjoint, parallel universes is why a local administrator who is added to the SharePoint farm administrators group after the original installation cannot deploy solutions -- the user account cannot get from the SharePoint universe to the SQL Server universe.

Visual Studio 2010 Deployment

Visual Studio 2010 has a great automatic deployment feature. The developer can right click a project to deploy to a designated web application and Site Collection. Unfortunately, in situations where the developer is not a local administrator, this fails. Why?

To answer that question, we need to understand what Visual Studio 2010 is doing during the deployment. With the Visual Studio 2010 solution open, display the project properties, select the SharePoint tab, and then click the View button in the Edit Configurations section. This lists the steps being executed on the right side under Selected Deployment Steps.

Let’s take the default steps one-by-one.

• Run Pre-Deployment Command:
  This is a non-issue. It will always work (barring pathological problems like running out of disk space or memory).
  
• Recycle IIS Application Pool:
  Must be a machine administrator. This is why you need to start Visual Studio 2010 as an administrator. This also prevents Visual Studio 2010 deployment if you don’t want the developer to be an administrator – unless you remove this step.
  You must also be a member of the dbowner role for the content database(s) associated with the application pool which hosts the target web application; that is, the site you specified when setting up the project. A simple way to do this is to run the cmdlet:
      Add-SPShellAdmin -user <domain\account> -database <content-database-guid>
  See the following PowerShell topic for details.
  
• Retract Solution:
  Member of the farm administrators group for farm solutions.
  Site Collection administrator for Sandbox solutions.
  
• Add Solution:
  Member of the farm administrators group for farm solutions.
  Site Collection administrator for Sandbox solutions.
  
• Activate Features:
  Depends upon the feature scope.
  Global (farm): farm administrator group member
  Web Application: farm administrator or member of Web Application full control user policy
  Site Collection: Site Collection administrator or owner
  Site: Site administrator or owner
  
• Run Post-Deployment Command:
  Again this is a non-issue. It will always work (barring pathological problems like running out of disk space or memory).
  

So we see Visual Studio 2010 deployment has stringent permission requirements. It is possible to work around some of these requirements by creating your own deployment steps, an exercise left for the reader.

PowerShell

Now we come to the often misunderstood cmdlet, Add-SPShellAdmin. Unfortunately, the Get-Help text does not clearly explain what this cmdlet does. It is not a general solution for SharePoint security permission issues. This cmdlet has a focused purpose; which is, to allow the designated user to run PowerShell commands that impact a particular database. Side effects are the user is also added to the database db_owner role and to the local machine WSS_ADMIN_WPG group, so again, be sure this is what you intend. This command does NOT add the user to either the local administrators groups or the Farm Administrators group; in other words, it grants least privileges.

If you execute the command WITHOUT a database parameter, the user will be able execute PowerShell commands that affect the Config database, but NOT content databases.

If you execute the command with a database parameter for a content database, the user will be able to execute PowerShell commands affecting that content database (and the config database).

To get the content database GUID to pass to the cmdlet, execute the Get-SPDatabase cmdlet. The following example shows UserNo115 granted permission to run PowerShell commands against the configuration database, and UserNo116 granted permission to run PowerShell commands against the WSS_Content_My content database.

The following 2 screen shots show the result of these commands as view through the SQL Management Studio. Note both users are now members of the Config database db_owners role; but only UserNo116 has access to the My Site content database.

What Can the User Do?

A useful troubleshooting tool is the command whoami. Logon to the server with the user’s credentials (or start a command prompt as the user), then open a command prompt and enter whoami /all. This will show the user’s group memberships and explicit privileges on the server. If the user is not in the expected groups, review the prior information to determine what went wrong. You should very rarely have to manually add users to groups except for Remote Desktop Users group.

The following output is for an administrative user, as seen by membership in the BUILTIN\Administrators group.

This example is for a user added to the farm administrator group through Central Administration. Note the absence of membership in the local administrators group, which is by design.

What about WSS Local Groups?

You may have noticed this discussion has not mentioned local groups like WSS_ADMIN_WPG, WSS_RESTRICTED_WPG_V4, and WSS-WPG. This was intentional. Membership in these groups is maintained automatically by Central Administration pages and PowerShell commands.

If you feel it necessary to modify these group memberships, my advice is STOP, take a deep breath, and review your overall approach to see what’s wrong with your security design. Modifying these groups’ membership is like putting a Band-Aid on a gushing wound. You have bigger problems.

Many people see error messages when attempting to create an SPDiag snapshot. In my experience the root cause is either the PowerShell remoting configuration or the credential delegating group policy. Sometimes this is not obvious if you only see a vague, generic error.

First, consider what happens during a snapshot. There are several steps, as implied by the TechNet description (http://technet.microsoft.com/en-us/library/hh144782.aspx)

You can take snapshots of your farm that aggregate

• report images
• farm topology information
• Unified Logging Service (ULS) logs
• usage database data.

This makes it easy to consolidate key troubleshooting information about a SharePoint farm and share it with other users or preserve it for comparison and trend analysis.

Errors can arise at any of these steps. As far as I know, you cannot select which steps you want executed other than selecting Light or Full in the Snapshot diaglog.

Report images seem a safe operation. SPDiag is merely turning the report into a picture and storing it in the Export Directory.

The farm topology step is where I see the most errors. To gather the farm topology, SPDiag establishes a remote PowerShell session to each server in the farm, including the database server. If either the PowerShell remoting configuration or the credential delegating group policy is not configured correctly, the snapshot will abort at the first error. You can still find what was collected up to that point in time in the export directory, but no further information is collected. In this scenario you will see the following dialog.

The first question is, where is the error.log file? It is in the export directory. Sometimes you will find a very explicit error description like the following:

05/19/2012 10:39:20       Connecting to remote server failed
with the following error message : The WinRM client cannot process the request.
A computer policy does not allow the delegation of the user credentials to the
target computer. Use gpedit.msc and look at the following policy: Computer
Configuration -> Administrative Templates -> System -> Credentials
Delegation -> Allow Delegating Fresh Credentials.  Verify that it is
enabled and configured with an SPN appropriate for the target computer. For
example, for a target computer name "myserver.domain.com", the SPN
can be one of the following: WSMAN/myserver.domain.com or WSMAN/*.domain.com.
For more information, see the about_Remote_Troubleshooting Help topic.

The first step is to verify the PowerShell configuration on the SharePoint farm servers. Open up this page, SharePoint Diagnostic Studio 2010 (SPDiag 3.0). It has the installation requirements and instructions.

We are interested in target server configuration. On each server in the farm, including the database server, Open a PowerShell window as administrator, and run the commands listed in the TechNet article, repeated here:

• Enable-PSRemoting -force
• Enable-WSManCredSSP -role Server -force
• Set-Item WSMan:\localhost\Shell\MaxMemoryPerShellMB 1000
• Set-ExecutionPolicy RemoteSigned

Now we need to set the PowerShell configuration on the client server. While the TechNet article doesn't mention this, I have found by experience the client server is where you are running SPDiag, even if it is one of the SharePoint farm servers. In my case, SPDiag is installed on the Central Administration server, so I have to execute these PowerShell commands as administrator.

• Enable-PSRemoting -force
• Enable-WSManCredSSP -role Client -DelegateComputer “<target_computer>” -force

The question is, What is the correct target_computer value? For a true remote client, it is one or more of the SharePoint servers, like the Central Administration server; but what if you are running SPDiag on farm server? I used the farm server name where SPDiag is installed. In my case, I had to use the Central Administration server FQDN name, even though logically it is not remote from the farm (however, it is remote from the other servers in the farm). 

Please see Enable-WSManCredSSP for details of the command syntax and how to specify multiple target servers. Notice the command expects the FQDN server name(s).

Now that PowerShell is configured for remote execution; that is, SPDiag can now execute PowerShell scripts remotely on the other farm servers from the Central Administration server, we need to address the error message above which refers to credential delegation. What is this all about?

When PowerShell executes a script remotely, it has to execute on the remote machine within a user security context. SPDiag uses the security credentials you enter in the Credential Request prompt to establish the security context on the remote server.

If you enter the Farm Administrator credentials, then the Farm Administrator account has be explicitly configured for remote delegation to be able to execute the remote PowerShell scripts on the other farm servers. This is what the above sample error message is about. It is telling us PowerShell cannot run remote scripts because it has not been authorized to delegate the credentials to the target server(s). The fix is simple. Just follow the step in the error message.

Run the gpedit.msc management console by entering gpedit.msc in the Run box of the Start Menu.

When the console opens, navigate the tree structure to Computer Configuration -> Administrative Templates -> System -> Credentials Delegation -> Allow Delegating Fresh Credentials.

Select the Enable option, and then click the Show button to see the list of servers that can be delegated to.

When the Show Contents window opens, use the Add button to add SharePoint farm servers and the database server. Use the format shown; that is, wsman/ prefixed to the server's NetBIOS name.

Note: It appears SPDiag expects the database server to be a single machine; or the currently active mirror. SPDiag (in my experience) is unable to resolve the logical name of clustered servers. In other words, SPDiag expects the name entered to be an individual server machine, not a cluster name. Hopefully future versions of SPDiag will be able to gracefully handle SQL clusters.

That's it. You should now be able to generate snapshots.