How many times have you heard the above from the Storage team, having looked at SQL's wait types and seen PAGEIOLATCH_*?

You know there's something gone wrong with your disks, because SQL's telling you so. But the guy/gal in the disk team isn't having any of it.

What next? How do you tell who's right? This is an amazingly common scenario, one which I've seen a hundred times (and have become embroiled in most of them).

This blog post will show you how to go about figuring out, from a SQL Server perspective, whether the disks really are at fault or if there's something important you've missed. This is going to be a pretty long haul but it's worth it, so get a cup of tea/coffee/local equivalent and get comfy.

First off we need to make some clarifications on terminology. This may seem academic to start with, but it's really important to get specific with your definitions. You'll see why shortly.

SQL's telling you PAGEIOLATCH_* is the major wait type. What does this really mean? It means the query tried to read a row from a page which was trying to do some physical IO (either reading or writing), and you had to wait on the latch to get to the page.

Note at this point; you're waiting for I/O to happen. The I/O stack in SQL is pretty shallow because we farm out IO requests to Windows, which in turn farms it out to the Kernel, which pushes it through a driver, which goes through the HBA (or equivalent), through the fibre, onto a SAN (probably), then through the various levels in the SAN, until it hits one or more spindles.

Hopefully at this point you're beginning to think: "Wow, that's a lot of things that can go wrong!". Indeed it is - each one of these layers can experience issues or contention. So rather than thinking "It's the disks!", we need to start thinking "It's the physical IO stack!".

Ok, so, we've got "physical IO contention". How does this redefinition help?

We can now approach troubleshooting the stack (rather than the disks) to see where the contention/problem lies. First off we need to figure out whether we're driving the stack too hard. The back end has a limit to the amount of data it can shift and at what latency, so let's make sure we're not pushing it too hard.

How hard is "too hard"? The answer may not be as simple as it first appears.

When one configures a SAN for SQL it's a good idea to run SQLIO to gauge what the physical IO stack is capable of doing in that configuration, to determine if that suits the database activity that will be occurring. If we're running a Data Warehouse we'll want to do lots of parallel scan operations on the data files (probably reads) and a fair amount of sequential writes on the logs. If we're running OLTP then we're likely doing lots of short, serial, random IO on the datafiles (reads and writes), and lots of short sequential writes to the logfiles. Same goes for TempDB.

So once the back end is correctly configured for the activity type and SQLIO has been run to check performance, we know at what throughput &/ latency the configuration tops out at.

If we ran SQLIO on the new system, and kept the output files and PerfMon output handy, we can refer to this data to find out the configuration’s maximum capabilities. We can then compare it to the current response patterns to see if the disks are being driven close to their known maximum. If we didn’t run SQLIO, then we need to just look at how the system is responding now, and judge if the system is overstressed.

PerfMon shows how the physical IO stack is responding. But what counters are best? For a start, NEVER use the Physical Disk counter unless we're connecting directly to physical disks. If a single drive in Windows maps to multiple disks then use Logical Disk. The reason for this is that many SANs hide the true number of physical disks from you, which makes conclusions confusing or downright incorrect.

These counters are your new Best Friends:

Logical Disk :: %Idle Time

Logical Disk :: Avg. Disk sec/Read

Logical Disk :: Avg. Disk sec/Write

Logical Disk :: Disk Read Bytes/sec

Logical Disk :: Disk Write Bytes/sec

SQL Server:Buffer Manager :: Page Life Expectancy

These counters should be captured over at least 20 minutes with 1 second interval for each disk, when the disks are under their peak load. I would normally expect to see a separate drive letter/mount point for System Databases, tempdb, Data files, Log files, and Backups. Each of these may have to have a different configuration to handle the differing file activity types (there’s a blog post coming up on this soon!).

"% Idle Time" tells us how hard the drive is working. If this counter is reading under 80% average or spiking under 40% (min) then the underlying disks are working too hard.

"Avg. Disk sec/Read" tells us the read latency for the drive. For the data file drive this should average under 17ms with no spikes over 25ms, for the TempDB drive it should average under 10ms with no spikes over 10ms.

"Avg. Disk sec/Write" tells us the write latency for the drive. For the datafile drive this should average under 17ms with no spikes over 25ms, for the TempDB and logfile drives it should average under 10ms with no spikes over 10ms.

"Disk Read Bytes/sec" gives us context for the above counters - it tells us the volume of reads happening on the drive. There is no "correct" value for this, but note that drives tuned for OLTP may not be able to provide high volume throughput.

"Disk Write Bytes/sec" gives us context for the above counters - it tells us the volume of writes happening on the drive. There is no "correct" value for this, but note that drives tuned for OLTP may not be able to provide high volume throughput.

"Page Life Expectancy" tells us how volatile the Buffer Pool is. If this is below threshold then we're probably doing too much reading and stressing the physical IO stack. See my previous blog on how to calculate PLE (https://blogs.msdn.com/b/mcsukbi/archive/2013/04/12/sql-server-page-life-expectancy.aspx).

Armed with the above counters we can tell if the issue that is happening can be dealt with by the SQL team, or by someone else.

If we have low idle time, high latency, and low throughput then the back end may be saturated with other people's activity, or incorrectly configured for the load type we're placing it under. We're not asking much of our drive but it's not responding as we'd expect. Check Page Life Expectancy (PLE) to see if the system needs tuning.

If we have high idle time, low latency, and high read/write throughput, we're doing a lot of work but the stack can cope. The PLE will likely be low but we need to find out what's running to cause such a large amount of write activity.

If we have high idle time, high latency, and low throughput, we're bottlenecking outside the disks. We need to check the driver levels and the HBA configuration. If these are good, check SAN fabric/node saturation levels.

There are too many specific combinations to cover here, but by now you should be getting the picture.

So in summary – if we suspect that the physical IO stack isn’t responding as we’d expect, make sure that the issue doesn’t lie a little closer to home.

Page Life Expectancy (PLE) is the best indication of how volatile your Buffer Pool is (BP). It's a PerfMon counter, found in the SQL Server:Buffer Manager PerfMon object.  Monitor it every 3-5 seconds or so. There is also the Buffer Node:Page Life Expectancy counter which should be considered for NUMA systems, using the same logic per node as one would on a non-NUMA system.

Volatility is measured by taking the average "life" of a page within the Buffer Pool (in seconds). If a page is overwritten or aged out, it starts a whole new life.

So if lots of pages are being overwritten with new data very often, the average PLE will be low, and our BP volatility will be high.  

Conversely if most pages in our BP remain there for a long time without being overwritten, the average PLE will be high, and the BP volatility will be low.

So why do we care about BP volatility and PLE? What can knowing the BP volatility do for us? What is a "good" and "bad" PLE figure?

PLE can be a measure of how much physical IO your SQL Server is doing. Hopefully I've got your attention, because physical IO is a major performance concern, both for reading and writing.

Let's say you're loading a large amount of data into SQL Server. You want this to get into SQL Server as quickly as possible, so you're using SSIS with lots of parallel threads, hash partitions on the table, no indexes, and all the other tricks described here: https://msdn.microsoft.com/en-us/library/dd537533(v=sql.100).aspx.

Because nothing goes onto the page without first going into the BP, your BP will be flooded with this new data. Your BP pages are being overwritten by as much physical IO as possible on the read, therefore your PLE dips sharply. If you're doing a big data load you want your PLE to be as low as possible, because this means the data load is going quickly.

Now let's say you're running a big report using Reporting Services. We don't want to do much physical IO (if any) because having to read data pages from disk will slow us down. In this case we want our BP volatility to be as low as possible, and our PLE to be as high as possible (or as high as necessary for the query to hit SLA, see future post on query SLA's).

So depending on the activity on the server, PLE can give us an indication of how efficient our physical IO/BP ratio is.

We therefore have to be aware of the dominant activity type on the server before we make a call on whether PLE is "good" or "bad".

The old recommendation was that PLE should have a minimum of about 300 seconds. This was from the days when SQL's BP was around 4GB or so. This therefore meant that for a read-mostly activity such as a report a PLE of 300 meant that the SAN was reading 4GB over 5 minutes, which calculates to about 3.4MB/s.
These days we have BP's around the 64GB and above. So our 300 second threshold now means that the SAN would be reading about 218.5MB/s, which is a fair amount and likely to cause comment!

So what's a "good" PLE for a read-mostly operation like a report? Here's a handy formula I use to get a decent estimate:

PLE threshold = ((MAXBP(MB)/1024)/4)*300

Where MAXBP(MB) is the maximum amount of Buffer Pool memory (in MegaBytes) over the report run you're interested in. You can use PerfMon to trace the maximum value of SQL Server:Buffer Manager:Database Pages over your reporting run.

Take that number of pages and convert to MB: (pages*8)/1024 then substitute that for ‘MAXBP(MB)’ in the above formula to get your minimum PLE.

The above formula considers each 4GB of BP (per NUMA node or otherwise) to have it’s own 300 second PLE. Therefore the more BP you have allocated, the higher the PLE threshold.

If you want a "quick and dirty" way of calculating the PLE threshold then you can use the sp_configure "max server memory (MB)" value in the formula above, but this isn't really accurate because it doesn't account for stolen memory (memory taken from the BP, usually for plan caching, compilation, optimisation etc), and is even less accurate for SQL Server 2012 because we have lots of new stuff considered in the "max server memory (MB)" setting for 2012.

If you're reading this because you're just after a threshold for PLE and you either know how to investigate low PLE, or you don't care, you can stop reading now.

However if you're interested in what to do when PLE crashes, read on!

PLE is an amazing counter - you can use it to quickly direct you straight towards the problem areas if you know a few little tricks.

If PLE is low there must be a reason for this - knowing the reason is the key to investigating a performance problem through PLE.

If you're doing a big load, you want PLE to be low, so that's fine. If we're doing read-mostly activities and it's under the threshold, here's the top 3 reasons:

1) The queries being run have changed to look at data that they weren't before, hence we now read in this new data into the BP. Or you're reporting just after a data load of different data, or just a large data load.

2) The queries being run are inefficient, and are reading unnecessary data into the BP which is therefore being constantly turned over

3) There are large numbers of efficient queries (or a few very large efficient queries) which are saturating the BP

Here's what you do for each of these situations

1) This is normal, but it should be short-lived. You'd expect to see the PLE drop off a cliff, then build back up over the threshold and stay there until the next shift in data requirement. If it drops and stays there, you're probably in situation 2 or 3

2) You can find this out by looking at the top queries for physical IO usage and checking out the query plans. If you see lots of scans, there may be some tuning to be done! Since 80% of performance problems come down to poor T-SQL (which may therefore create inefficient Query Plans), consider this before you think you're in situation 3!

3) You've done your best, the queries are as efficient as they can get, but your PLE is still low. You can either spread out the queries (maybe running Agent Jobs one after the other instead of at the same time), or increase the amount of Buffer Pool memory. NEVER leave less than 1GB free for the OS, 2GB for servers with >64GB RAM, 4GB for servers with >128GB RAM. It's not that it's "free" and never used, the OS will use it and release it in much less than a second - less time than PerfMon or most tools' data gather interval, so it's being used but you don't get to see it. So if your PerfMon counter "Memory:Available Memory in MB" is under these thresholds and you're in situation 3, you need more physical memory so you can increase BP AND keep enough memory for the OS.

So there's PLE in a nutshell, look out for the next blog post on how to convince your Storage team that the Storage isn't working properly - I'll show you how to provide unequivocal proof, or how to tell if you're barking up the wrong tree! 

I have recently been exploring the use IaaS (Infrastructure as a Service) to provide cloud-based virtual machines (VM’s) as opposed to laptop-based VM’s and also PaaS (Platform as a Service) for SQL databases.  I like the idea of carrying around a lighter more portable laptop and using cloud services to help me day to day, in contrast to carrying a heavy weight workstation for Hyper-v usage.

I know we can deploy VM’s through the Azure Portal but I prefer an automated approach.  Fortunately the new PowerShell cmdlets support Azure VM provisioning and also Azure SQL database provisioning (plus some other nice interfaces).  This enables me to quickly spin up a SQL Server VM in Azure or SQL database in Azure.   I am actually quite amazed what is possible with PowerShell and it is my new best friend.  PowerShell ISE in Windows 8 is superb, I highly recommend this as a development environment due to the intellisense and cmdlets search pane integration. 

Below is the PowerShell script I wrote to provision a VM.  Unfortunately some manual steps are still required if you wish to manage the SQL Server instance remotely through Management Studio e.g. opening firewall ports, enabling TCP etc.   This is all documented here https://www.windowsazure.com/en-us/manage/windows/common-tasks/sql-server-on-a-vm/.  Fortunately it is a lot easier to connect to an Azure SQL database and we can automate the registration of the provisioned instance in Management Studio.

Azure VM Provisioning (SQL Server 2012 Evaluation Edition)

Step 1. Download and register the Azure publishing certificate (one time only event). 

In order to use PowerShell with the Azure VM and SQL Database services you will need to download and import the publishing file. Fortunately, this is a simple process.  I also store my certificate on Skydrive so I can access it everywhere I go in case I need it again.

Get-AzurePublishSettingsFile
Import-AzurePublishSettingsFile C:\...

If you don’t import the publishing file then you may see an error similar to below when attempting to access the Azure services.

An error occurred while making the HTTP request to https:

//management.core.
windows.net/ae81ecb1-a8af-4fb7-87c5-4418babb4ff2/services/sqlservers/servers
/<server>?op=ResetPassword. This could be due to the fact that the server 
certificate is not configured properly with HTTP.SYS in the HTTPS case. This 
could also be caused by a mismatch of the security binding between the client 
and the server. 

Step 2. View available subscriptions and set the correct Subscription

Get-AzureSubscription | Select SubscriptionName
Select-AzureSubscription –SubscriptionName

Incidentally, you may also need to associate a specific Azure Storage Vault (ASV) Account with your subscription, for some reason the default value was null so I had to allocate a specific account.  The Azure Storage account is required to host the VM disks which are provisioned during the creation of an Azure VM image.  

Get-AzureStorageAccount | Select Label
Set-AzureSubscription -SubscriptionName "Windows Azure MSDN - Visual Studio Ultimate" 

-CurrentStorageAccount "<storageaccount>"

Step 3. View the available Azure VM images and locations

Get-AzureVMImage | Select ImageName
Get-AzureLocation | Select DisplayName

Step 4. Create an Azure VM

I like my VM’s big! ExtraLarge!  You can use the New-AzureQuickVM syntax, New-AzureVM or New-AzureVMConfig syntax,  The New-AzureQuickVM automatically creates and provisions the VM which does not require any additional steps. 

New-AzureQuickVM -Windows -ServiceName "<azureservice>" -Name "<vmname>" 
-ImageName "b83b3509582419d99629ce476bcb5c8__Microsoft-SQL-Server-2012-
Evaluation-CY13Feb-SQL11-SP1-CU2-11.0.3339.0" –Password <password> -Location "North Europe" 
–InstanceSize “ExtraLarge” 

You should see something like this below. 

If you see a DNS error then you may be trying to provision a under a duplicate service name e.g. Error "DNS name already exists" is misleading as it refers to a duplicate service name so change this in the parameters. This servicename refers to the VM service by which you reference/connect e.g. <vmservice>.cloudapp.net

Step 6. Start the Azure VM

Start the Azure VM using the command below.

Start-AzureVM -ServiceName "<servicename>" -Name "<vmname>"

You can view the properties of your Azure VM’s by using Get-AzureVM –ServiceName <ServiceName>.  I would also add that additional data disks can be simply added using the Add-AzureDataDisk syntax providing the ability to simply increase the capacity of the provisioned Azure VM instance.

Step 7. Download the  remote desktop connection file to your local desktop!

Another great feature is the ability to automatically download the remote desktop file for the provisioned Azure VM. 

Get-AzureRemoteDesktopFile -ServiceName "<ServiceName>" -name "<vmname>" -LocalPath "$ENV:userprofile\Desktop\myVm01.rdp"

It is also possible to automatically launch and connect to your VM instance using the standard Remote Desktop client, mstsc.exe, which just eliminates another step in the process to connect to your Azure VM. 

mstsc $ENV:userprofile\Desktop\myAzureVm1.rdp

Of course you could parameterise all of this to make life even simpler. 

Step 8. Connect!

Simply enter the username and password off you go, simple.   You could create multiple PowerShell batch files for each Azure VM image type.  I am sticking with the SQL Server 2012 instance for now. 

Teardown

Cleaning up the environment is simple too, just a couple of PowerShell commands to stop and remove the provisioned VM.

Step 9. Stop the Azure VM

Stop-AzureVM  -ServiceName "<ServiceName>" -Name "<VmName>"

Step 10. Remove/delete the Azure VM

Remove-AzureVM -ServiceName "<ServiceName>" -Name "<VmName>"

Step 11. Remove/delete the Azure VM disks

The VHD’s associated with the image are not automatically removed so you will need to issue the Remove-AzureDisk command.  You can view the existing VHD’s and the associated image and container using the Get-AzureDisk command as shown below.  You will notice that I only have one disk (VHD) associated to an image.  The other VHD’s were from previous Azure VM deployments. 

Removing (deleting) the VHD is simple.  The –DeleteVHD parameter is required if you wish to permanently delete the image from ASV so use with caution!

Remove-AzureDisk –DiskName <diskname> –DeleteVHD

Step 12. Remove the allocated cloud service

Remove-AzureService -ServiceName "<servicename>" 

Step 13. Remember to save your PowerShell script and parameterise it for one-click deployment!

Azure SQL Database Provisioning

On a related noted, it is also possible to provision a Windows Azure SQL Database using the new PowerShell cmdlets allowing me to rapidly deploy a cloud-based relational data store.  You must register the Azure publishing certificate unless this has been done previously (as above in step 1.).

Import-AzurePublishSettingsFile C:\....
Set-AzureSqlDatabaseServer –ServerName <server> –AdminPassword <password>

Creating a new Azure SQL Database instance is also easy, an example is provided below.  I chose North Europe due to my geographical location. 

New-AzureSqlDatabaseServer -location "North Europe" -AdministratorLogin "<login>" 
-AdministratorLoginPassword "<password>” 

You’ll need to create a firewall rule so you can connect to the new Azure SQL Database instance:

New-AzureSqlDatabaseServerFirewallRule -ServerName <server> –RuleName 

<rulename>

 
-StartIPAddress "0.0.0.0" -EndIPAddress "222.222.222.222"

As I was working through this, I discovered a VERY useful command which pops up a dialog with the help options for a specific command, an example is shown below.  Omitting the –ShowWindow syntax will output the help details to the console window.

Get-Help Set-AzureSqlDatabase –ShowWindow

The next step was to create a SQL Server authenticated connection to the server hosting the Windows Azure SQL Database.  This is an important step as it establishes the context for the connection. 

#specify sql auth credential
$servercredential = new-object System.Management.Automation.PSCredential("<username>", 
("<password>" | ConvertTo-SecureString -asPlainText -Force))

#create a connection context
$ctx = New-AzureSqlDatabaseServerContext –ServerName <servername> -Credential $serverCredential

Incidentally, if you are wondering what is stored in the connection context then see below:

ServerName        : <server>
SessionActivityId : 850d5e6f-7201-4bad-8fd5-331086064d4a
ClientSessionId   : e8d82a6d-0ed2-4aa3-9c38-3c3da924ab6a-2013-03-13 15:49:35Z
ClientRequestId   : d2436b67-ecca-453d-8d7c-12619e599784-2013-03-13 16:02:36Z
Databases         : {master} 

Wondering what databases you can see?  Easy.

Get-AzureSqlDatabase -Context $ctx

Which returns all the databases deployed on the provisioned instance.  In the case below, only the master database was listed as no other databases are currently deployed.

Name          : master
CollationName : SQL_Latin1_General_CP1_CI_AS
Edition       : Web
MaxSizeGB     : 5
CreationDate  : 12/03/2013 22:35:57

Want a new database?  Easy again.

New-AzureSqlDatabase -Context $ctx –DatabaseName <databasename>-Collation SQL_Latin1_General_CP1_CI_AS 
-Edition Web -MaxSizeGB 5

Interestingly, there are a host of DataServiceContext class options made available under the context of the connection such as ServerMetrics and DatabaseMetrics.  This provides some interesting insight into the metadata for your Azure SQL database server such as throttled connections and failures.   Unfortunately, the context commands are not documented right now so this is just exploratory and the exposed properties may be removed in the future. 

$ctx.ServerMetrics.IncludeTotalCount()
$ctx.DatabaseMetrics.IncludeTotalCount()

Beyond the ability to provision a Windows Azure SQL Database using PowerShell cmdlets, I can also save time by automatically registering the Azure SQL instance in SQL Server Management Studio by invoking the SQL Server 2012 PowerShell command New-Item as below (thereby saving even more time!).  The AzureSqlDbServer1 reference is the friendly name which appears in the SQL Server Management console.

Import-Module sqlps
Cd "sqlregistration\Database Engine Server Group"
New-Item AzureSqlDbServer1 -ItemType Registration -Value "server=<server>.database.windows.net; 
integrated security=false; userid=<username>; password=<password>; initial catalog=<databasename>"

Teardown

Removing (or de-provisioning) the Azure SQL database, instance and Management Studio registration is simple.  The last command, Remove-Item, is a SQL Server PowerShell command to delete the Management Studio server registration and this must be invoked using sqlps as above.

Remove-AzureSqlDatabase $ctx –DatabaseName "<dbname>"
Remove-AzureSqlDatabaseServer -ServerName "<AzureSqlDbServer>"

Import-Module sqlps
Remove-Item AzureSqlDbServer1

Closing Remarks

The new PowerShell cmdlets for Azure are a fantastic way to easily provision either VM’s or a database in the cloud.  I will be parameterising my scripts (and including try.. catch blocks) to quickly create an Windows Azure Virtual Machine or Windows Azure SQL Database as needed (one-click deployment made easy!).  PowerShell ISE is also an excellent development environment which can be leveraged for not only Azure VM or SQL database provisioning but also for many more solution scenarios. 

What is missing?

• Provisioning Azure HDInsight clusters is not currently possible however this should be coming soon https://hadoopsdk.codeplex.com (refer to Programmatic Cluster Management). 
• PowerShell cmdlets for Azure SQL Reporting are not currently available.
• The ability to provision an Azure VM image with the full business intelligence stack deployed i.e. SharePoint 2013, PowerView and Power Pivot integration. 
• PowerShell remoting to be automatically enabled in the Azure VM.
• The ability to invoke SqlCmd and query Azure SQL databases through PowerShell (inc. support for Federations). 

References

Managing Windows Azure SQL Databases with PowerShell https://gallery.technet.microsoft.com/scriptcenter/Managing-Windows-Azure-SQL-632acc4b

Windows Azure SQL Database Management with PowerShell(https://blogs.msdn.com/b/windowsazure/archive/2013/02/07/windows-azure-sql-database-management-with-powershell.aspx

Getting Started with SQL Server on a Windows Azure Virtual Machine https://www.windowsazure.com/en-us/manage/windows/common-tasks/sql-server-on-a-vm/

Over the past couple of months I have occasionally been losing my PowerPivot Slicers in Excel when using the PowerPivot 2012 add-in.  I have now managed to put together a repro for the issue and will outline this below.

The issue has been reported via Connect, so should receive attention soon.  Until that point, this blog post provides my own (unofficial) description of the problem and how to avoid it, in case other people are hitting it.

Symptoms

The workbook will initially work as normal, i.e. slicers appearing and functioning as normal:

At some point later (maybe hours, days or weeks), you will re-open the workbook and the slicers will appear very broken. Sometimes they disappear completely, at other times they change size.  They often get dropped from the "Slicers Vertical" and "Slicers Horizontal" sections in the PowerPivot Field List, e.g.

Unfortunately the issue is present in both the RTM and CU2 releases of the add-in.  (I have only tested the RTM x86 and CU2 x86 add-ins so cannot definitely say if the issue is present in other releases).

Detailed Reproduction Steps

1. Double click a PowerPivot workbook in Windows Explorer to open it in Excel.
2. DO NOT click on any of the charts / pivot tables.
3. Click on the PowerPivot window button in the ribbon.
4. Refresh all (in the PowerPivot window).
5. Refresh the Pivot Chart (right mouse click on the chart, refresh).
6. Save the workbook and close Excel.
7. Reopen the workbook.
8. Click the pivot chart (to load the PowerPivot data).
9. Click off the chart.
10. Click back on it, the slicers have now mysteriously become disconnected.

Workaround

The clue to avoiding the issue is in step 2 above...i.e.

Ensure the PowerPivot Model is loaded into Excel before opening the PowerPivot window to refresh.  Or put even more simply...

ALWAYS CLICK ON A PIVOTTABLE, PIVOTCHART OR

SLICER BEFORE OPENING THE POWERPIVOT WINDOW

And, with that, you should be able to continue to use your Slicers.

Updates

More details (and probably future updates to this issue) will be found on Connect:

https://connect.microsoft.com/SQLServer/feedback/details/755850/bug-powerpivot-2012-slicers-are-lost-from-powerpivot-field-list#details

The Connect item also includes example files plus a short video to repro the issue.

In some environments it is not possible to install Dashboard Designer using the normal process (i.e. allow it to install automatically when first opened from SharePoint).  Some Citrix environments for example have permissions that prevent ClickOnce apps from being able to install. 

In these cases people sometimes look for alternative methods (e.g. using an installer / MSI).  Unfortunately, there is no MSI available for download for Dashboard Designer from Microsoft.  However, it is possible to put together an alternative method for installation.  The approach outlined below is one approach I have used in a couple of customer environments.

The process is in two parts:

1. Gather the files to install Dashboard Designer
2. Install Dashboard Designer

Note – This is not an officially supported way of installing Dashboard Designer, but it has worked where I have used it in the past.  If thinking of using this, you should thoroughly test it in your environment as your environment may throw up issues I haven't encountered.  Also, when updates to Dashboard Designer are released (e.g. in SharePoint Service Packs) it will be necessary to repeat this alternative installation procedure to get the updated versions deployed.

Pre-reqs

This process does not install the pre-reqs for Dashboard Designer:

• .NET Framework 3.5 Service Pack 1
• Microsoft® SQL Server® 2008 R2 Native Client – Download
• Microsoft® SQL Server® 2008 R2 ADOMD.NET – Download

You should ensure these are also deployed to the target environment prior to installing Dashboard Designer.

These pre-reqs apply irrespective of whether you are installing using the normal ClickOnce method or an alternative approach (the ClickOnce installation does not install these pre-reqs).

Gather the files to install Dashboard Designer

This approach requires a machine* that you can install Dashboard Designer on in the normal way (i.e. download the Click Once app from SharePoint).  On this machine, we’ll grab a copy of the Dashboard Designer files, ready to install on another machine (i.e. where the Click Once apps don’t work).

(*This machine also needs Visual Studio or another .NET SDK on to be able to use mage.exe).

1. On a machine with Visual Studio installed, run
   mage.exe –cc
   This clears the click once cache.  This means that after step 2, the only files in the Click Once cache are the Dashboard Designer files.
2. Open Dashboard Designer from SharePoint (i.e. allow it to install using the normal Click Once install mechanism).
   This downloads Dashboard Designer and installs it.  The dashboard designer files are copied into the Click Once cache, where we can pick them up.
3. Go to the Click Once cache - for my profile this is:
   C:\Users\xchrisbailiss\AppData\Local\Apps\2.0\...
4. Copy all of the dll, exe and config files out of this directory and subdirectories (ignore all the manifests, etc). 
   The easiest way to locate the files (since they are scattered across various subdirectories) is to search and specify the wildcard character *
   A whole bunch of directories etc will be listed.  Copy the dll / exe / config files.  For those files in an En folder, copy these into an En subdirectory.
   Some files appear to be present in multiple directories on some installations.  These files have the same name, size and last modified date so are very likely just multiple copies of the same file.
5. Check the file and folder structure against the file list below:
   1. Most of the files are in a single folder
   2. Inside this folder is a single subfolder named "En" which some of the files sit in.

File list

• DashboardDesigner.exe
• DashboardDesigner.exe.config
• DashboardDesigner.resources.dll
• Microsoft.PerformancePoint.Common.Calculation.dll
• Microsoft.PerformancePoint.Common.Calculation.resources.dll
• Microsoft.PerformancePoint.Scorecards.Client.dll
• Microsoft.PerformancePoint.Scorecards.Client.resources.dll
• Microsoft.PerformancePoint.Scorecards.Common.dll
• Microsoft.PerformancePoint.Scorecards.Common.resources.dll
• Microsoft.PerformancePoint.Scorecards.Designer.Framework.dll
• Microsoft.PerformancePoint.Scorecards.Designer.Framework.resources.dll
• Microsoft.PerformancePoint.Scorecards.DesignerPlugins.dll
• Microsoft.PerformancePoint.Scorecards.DesignerPlugins.resources.dll
• Microsoft.PerformancePoint.Scorecards.DesignerWorkspace.dll
• Microsoft.PerformancePoint.Scorecards.DesignerWorkspace.resources.dll
• Microsoft.PerformancePoint.Scorecards.WizardFramework.dll
• Microsoft.PerformancePoint.Scorecards.WizardFramework.resources.dll
• Microsoft.SharePoint.Client.dll
• Microsoft.SharePoint.Client.Runtime.dll
• En\DashboardDesigner.resources.dll
• En\Microsoft.PerformancePoint.Common.Calculation.resources.dll
• En\Microsoft.PerformancePoint.Scorecards.Client.resources.dll
• En\Microsoft.PerformancePoint.Scorecards.Common.resources.dll
• En\Microsoft.PerformancePoint.Scorecards.Designer.Framework.resources.dll
• En\Microsoft.PerformancePoint.Scorecards.DesignerPlugins.resources.dll
• En\Microsoft.PerformancePoint.Scorecards.DesignerWorkspace.resources.dll
• En\Microsoft.PerformancePoint.Scorecards.WizardFramework.resources.dll

Install Dashboard Designer

We are now in a position to test the install.  Note - the files simply need to be copied to the target machine.

1. Copy the files in this structure to the target machine.  They can be placed anywhere (example install folder structure given below).  No further activities are required – the install is as simple as copying the files.
2. Run Dashboard Designer.
3. Enter the SharePoint URL when prompted, e.g. https://MySharePointServer/ 

On one machine I tested this on, the Designer crashed after the SharePoint URL was first entered.  Upon reopening however, the designer was fine (and the URL had been saved correctly).  This crash appeared to only happen on the very first time Dashboard Designer was opened – subsequently it always opened without any problem.

Install Example

Dashboard Designer can be installed to any folder. 

For the purposes of this example, assuming that we are installing to
c:\Dashboard Designer

So, simply copy the above files to the following paths (no MSI is needed, no DLL registration is needed):

• C:\DashboardDesigner\DashboardDesigner.exe
• C:\DashboardDesigner\DashboardDesigner.exe.config
• C:\DashboardDesigner\DashboardDesigner.resources.dll
• C:\DashboardDesigner\Microsoft.PerformancePoint.Common.Calculation.dll
• C:\DashboardDesigner\Microsoft.PerformancePoint.Common.Calculation.resources.dll
• C:\DashboardDesigner\Microsoft.PerformancePoint.Scorecards.Client.dll
• C:\DashboardDesigner\Microsoft.PerformancePoint.Scorecards.Client.resources.dll
• C:\DashboardDesigner\Microsoft.PerformancePoint.Scorecards.Common.dll
• C:\DashboardDesigner\Microsoft.PerformancePoint.Scorecards.Common.resources.dll
• C:\DashboardDesigner\Microsoft.PerformancePoint.Scorecards.Designer.Framework.dll
• C:\DashboardDesigner\Microsoft.PerformancePoint.Scorecards.Designer.Framework.resources.dll
• C:\DashboardDesigner\Microsoft.PerformancePoint.Scorecards.DesignerPlugins.dll
• C:\DashboardDesigner\Microsoft.PerformancePoint.Scorecards.DesignerPlugins.resources.dll
• C:\DashboardDesigner\Microsoft.PerformancePoint.Scorecards.DesignerWorkspace.dll
• C:\DashboardDesigner\Microsoft.PerformancePoint.Scorecards.DesignerWorkspace.resources.dll
• C:\DashboardDesigner\Microsoft.PerformancePoint.Scorecards.WizardFramework.dll
• C:\DashboardDesigner\Microsoft.PerformancePoint.Scorecards.WizardFramework.resources.dll
• C:\DashboardDesigner\Microsoft.SharePoint.Client.dll
• C:\DashboardDesigner\Microsoft.SharePoint.Client.Runtime.dll
• C:\DashboardDesigner\En\DashboardDesigner.resources.dll
• C:\DashboardDesigner\En\Microsoft.PerformancePoint.Common.Calculation.resources.dll
• C:\DashboardDesigner\En\Microsoft.PerformancePoint.Scorecards.Client.resources.dll
• C:\DashboardDesigner\En\Microsoft.PerformancePoint.Scorecards.Common.resources.dll
• C:\DashboardDesigner\En\Microsoft.PerformancePoint.Scorecards.Designer.Framework.resources.dll
• C:\DashboardDesigner\En\Microsoft.PerformancePoint.Scorecards.DesignerPlugins.resources.dll
• C:\DashboardDesigner\En\Microsoft.PerformancePoint.Scorecards.DesignerWorkspace.resources.dll
• C:\DashboardDesigner\En\Microsoft.PerformancePoint.Scorecards.WizardFramework.resources.dll

A Note About the User.Config File

This process does not create the user.config file.  This will normally be created automatically the first time Dashboard Designer is run.  Again, on some environments this may not be possible and creating it as part of the installation process may be desirable. 

When Dashboard Designer is installed using this alternative approach, Dashboard Designer will create this file at the following location (where in my case [username] is xchrisbailiss):

C:\Users\[username]\AppData\Local\Microsoft_Corporation\DashboardDesigner.exe_StrongName_wl4f4koizhxra1ieka4fa0ofam43g0lh\14.0.0.0

If creating this file manually, you also probably want to set the following settings within this file:

• “MruSiteCollection” to the SharePoint URL, e.g. https://MySharePointServer/
• “Default Workspace Location” to the users default documents folder.

Introduction

This post is part of a series of posts describing the authentication methods supported by the SharePoint 2010 Business Intelligence service applications.  Please see Part 1 for an overview of this series of posts.

This post summarises the previous six posts.

Summary

The following table summarises how each of the BI Service Applications can authenticate to a SQL Server data source (Relational Database or Analysis Services – see above for test cases):

Key:

• Yes - Service application supports the specified authentication mechanism.
• No - Service application doesn’t support (i.e. doesn’t work) when web application uses the specified authentication mechanism.
• Per User - Service application is able to delegate the user identity to the data source.
• Shared - Service application is able to use a single, shared, trusted account to connect to the data source.
• N/A - Authentication back to data source not applicable to service application

Notes:

• PerformancePoint Dashboard Designer does not work with Claims-mode web applications.  The web application must be extended to provide an alternative URL in classic-mode.
• Reporting Services BIDS Report Designer and Report Builder are not able to retrieve / save content into Claims-mode web applications.  The web application must be extended to provide an alternative URL in classic-mode.

Additional References

• Configure Kerberos Authentication for SharePoint 2010 – Whitepaper
• Implementing Claims-Based Authentication with SharePoint Server 2010 - Whitepaper

Introduction

This post is part of a series of posts describing the authentication methods supported by the SharePoint 2010 Business Intelligence service applications.  Please see Part 1 for an overview of this series of posts.

This post describes how to test that user identity is being delegated through the BI service applications in a web application utilising FBA-Claims.  It also covers some differences in the functionality that is supported by FBA-Claims.

BI Service Application Tests in the Claims Web App – FBA-Claims

For a third time I’ll walk through each of the BI Service Applications tested previously – again accessing them in the claims web app, only this time authenticating as a forms based user (FBA-Claim).

To state the obvious, FBA-Claims have no associated Windows identity.  This section will describe the consequences of this and how it can be worked around.

Excel Services

Accessing the test workbook created earlier in the Claims site allows the same test to be repeated:

C2WTS is now unable to obtain a windows security token and the connection to SQL Server fails.  There is therefore now no way to delegate the user identity to the back-end. 

It is still possible to connect to the back-end using a credential stored in the Secure Store Service.  This is configured on the connection properties in the Excel application:

Enter either a specific Secure Store Service ID (SSS ID) or select None to pick up a default credential configured in the Secure Store Service and Excel Services.  The workbook can then connect to the SQL Server database using the stored, shared credential:

PerformancePoint Services

Accessing the test dashboard created earlier now results in:

As the error message helpfully explains, it has not been possible (for the C2WTS service) to obtain a windows identity from the non-windows claim.

To work around this again requires using a shared credential stored in the Secure Store Service.  However, it is still possible to pass the identity into Analysis Services (albeit via a mechanism that is harder to work with in Analysis Services) using the PPS Data Connection Authentication option “Unattended Service Account and add authenticated user name in connection string”.

The dashboard then appears as:

The user name (in claims format) then appears in the Custom Data measure.

Reporting Services

Reporting Services was already configured in the previous section to work with a shared account.  This continues to work with FBA-Claims:

PowerPivot for SharePoint

As described previously, PowerPivot for SharePoint works only with Classic-mode authentication.

Visio Services

Accessing the test web drawing created earlier now results in an error:

Again, this is due to there being no windows identity to authenticate to SQL Server with.  Specifying a trusted shared account in the Secure Store Service allows the drawing to work, albeit again without the original user identity being delegated to the back-end:

It is worth noting that the Secure Store Service can only be used for web drawings that use an ODC file to specify the connection.

Is Claims Augmentation the Answer?

So, the FBA-Claims tests show that none of the key BI services are able to transition an FBA-Claim to a windows security token. 

This may seem surprising; since the C2WTS service can obtain a windows security token, provided it is presented with a UPN claim (see MSDN).   This offers the theoretical possibility that adding a UPN claim into the set of claims might allow C2WTS to magically make the BI services work.

Well, it is possible to add additional claims using a Claims Augmenter (see MSDN), including adding the UPN claim:

Unfortunately, the BI service applications will only invoke the C2WTS for Windows-Claims.  Each claim is tagged with the provider that issued it.  This tag is immutable, so you cannot create a UPN claim from a claims augmenter that will be usable by SharePoint.

So, no, claims augmentation is not the answer :-(

[Aside: The custom membership provider, custom role provider and claims augmenter were contained in a single assembly for development purposes.  This assembly was deployed into the farm using a wsp solution, which dropped it into the WFE.  It was necessary to also deploy it manually into the GAC on the the application servers – without this the service apps were failing on these servers].

Continued...

Continue reading in Part 7.

Introduction

This post is part of a series of posts describing the authentication methods supported by the SharePoint 2010 Business Intelligence service applications.  Please see Part 1 for an overview of this series of posts.

This post describes how to test that user identity is being delegated through the BI service applications in a web application utilising Windows-Claims.  It also covers some differences in the functionality that is supported by Windows-Claims (vs Classic-authentication).

BI Service Application Tests in the Claims Web App – Windows-Claims

I’ll now walk through each of the BI Service Applications tested previously – this time accessing them in the claims web app as a Windows User (i.e. using a Windows-Claim).

Excel Services

Uploading the test workbook created earlier into the Claims site allows the same test to be repeated:

This shows that we are able to delegate the user identity back to the SQL Server instances.  Behind the scenes, the C2WTS has successfully performed a protocol transition from a Windows-Claim to a Windows Kerberos Ticket (two in fact, one for the Relational Engine and another for Analysis Services).

PerformancePoint Services

The first challenge in working with PerformancePoint on a Claims-enabled site is opening Dashboard Designer.  Dashboard Designer (a Click-Once application) is not Claims-aware.  Attempting to open it on a Claims-enabled site (even using a Windows-Claims as here) results in:

This can be worked around by extending the web application (see Technet), to enable access via Classic-mode authentication.  In my lab environment, I extended the claims web app, to https://claimsext, from where Dashboard Designer can be successfully opened:

The same test dashboard as shown previously can then be built and published.  This can be viewed via either the https://claimsext URL (not shown) or https://claims URL (shown below):

Again, the highlighted cells show that the current connection from PPS has been authenticated using Kerberos and that the user identity has been delegated through to Analysis Services correctly.

Reporting Services

When opening Report Builder from a Claims-mode site, it is not possible to retrieve / save reports, data sources, etc from / into the SharePoint site.  Instead, the following prompt is thrown up:

It is impossible to enter a credential into this dialog – it simply won’t go away.

A similar prompt is thrown up when attempting to deploy into a Claims-mode site from Visual Studio.

As with PPS Designer, extending the web application to provider a Classic mode URL will enable both tools to access content in the site.

Uploading a copy of the report created earlier allows us to test running a report in Reporting Services.  However, attempting to pass the user identity to the back-end now results in an error, e.g. attempting to test the Data Source:

Windows authentication / per-user identity delegation is not supported in SQL Server Reporting Services 2008 R2 when the web application is in Claims Mode.  Instead, a shared account must be used (see MSDN).  That account will either be the default trusted execution account for the server (in which case the fall back will be transparent) or the account can be configured explicitly on the data source:

Of course, the disadvantage of using a shared trusted account is that the user identity is no longer passed through to the back-end:

This restriction is due to the fact SQL Server Reporting Services 2008 R2 is not claims-aware.  It runs as a separate web-server outside of SharePoint.  In SQL Server 2012 Reporting Services becomes a SharePoint service application and is better suited to supporting this type of scenario.

PowerPivot for SharePoint

Attempting to open a PowerPivot workbook from the Claims-enabled site initially appears successful – just after opening the workbook it appears as:

However, at this point, no PowerPivot functionality has been invoked, only Excel Services has been used, which is showing data that was cached in the worksheet.  The PowerPivot service application only comes into play when Data >> Refresh… is selected from the toolbar, or the slicer selection is changed.  Trying either of those actions results in:

At the current time, PowerPivot does not support Claims-enabled sites of any kind.  Only web applications running Classic-mode authentication are supported.

Visio Services

Uploading the test web drawing created earlier into the Claims site allows the same test to be repeated:

Again, this shows we have successfully delegated the user identity back to SQL Server.

Continued...

Continue reading in Part 6.

Introduction

This post is part of a series of posts describing the authentication methods supported by the SharePoint 2010 Business Intelligence service applications.  Please see Part 1 for an overview of this series of posts.

This post describes my claims-mode web application – “claims”.  The functionality of the BI services accessed via this web application are described in the following posts.

Claims Web Application

After getting the portal web application running, I configured a second web application – “claims”.  The claims web app uses the Claims-mode authentication provider.

The claims web app utilises the same set of service applications as the portal web app.  This means it will pick up the existing working set of service applications and their Kerberos-related configuration.

Note - even though this web application uses claims, Kerberos is still relevant.  This is because the back-end SQL Server instances are not claims-aware – they are geared to work with windows authentication. For outbound authentication to the back-end, SharePoint uses the Claims to Windows Token Service (C2WTS), part of the Windows Identity Foundation, to obtain a windows identity (Kerberos ticket) from the set of claims.  To do this, the C2WTS performs a Kerberos Constrained Delegation Protocol Transition – hence Kerberos is still relevant.

The claims web app supports both Windows-Claims and Forms Based Authentication (FBA-Claims).

You may expect that a user authenticated with a Windows-Claim will be able to access the same functionality from the SharePoint BI service applications as a user authenticated into the Portal web app using Classic-mode windows authentication.  This is in fact not always true and varies by service-application as I’ll describe in the next couple of posts.

FBA-Claims do not have a windows identity associated with them, so will obviously not be able to delegate a windows identity.  Could C2WTS help here?  Not really, as I’ll also describe in a later post.

Upon navigating to the claims URL, the default screen is presented where the user is asked to select the authentication method:

If Windows Authentication is selected, the user is automatically logged in based on a Windows-Claim based on their active directory account.

If Forms Authentication is selected, the user is redirected to a second login screen:

These FBA tests used a simple custom membership provider and a simple custom role provider.  References:

• System.Web.Security.MembershipProvider Class - MSDN
• System.Web.Security.RoleProvider Class - MSDN
• Claims Walkthrough: Creating Forms-Based Authentication for Claims-Based SharePoint 2010 Web Applications Using Custom Membership and Role Providers – MSDN

A simple Claim Viewer web part was also created, which lists the users claims.  A sample of a similar viewer can be found at MSDN. 

When logged in with a Windows Claim, the following set of claims is displayed:

The UPN claim (https://schemas.xmlsoap.org/ws/2005/05/identity/claims/upn) is worth noting.  The UPN claim is used by the Claims to Windows Token Service (C2WTS) to obtain a windows security token that can be delegated to systems that aren’t claims aware such as the SQL Server Relational Database Engine and Analysis Services.

When logged in with an FBA Claim, the following set of claims is displayed:

Continued...

Continue reading in Part 5.

Introduction

This post is part of a series of posts describing the authentication methods supported by the SharePoint 2010 Business Intelligence service applications.  Please see Part 1 for an overview of this series of posts.

This post describes how to test that user identity is being delegated through the BI service applications in a web application utilising classic-mode authentication.

BI Service Application Tests in the Portal Web App – Classic-Authentication

Let’s quickly walk through some authentication tests for each service application. 

Note, for brevity, the tests illustrated here don’t cover all possible delegation paths (extend the tests in your own time, for example, to test delegation via PerformancePoint Services to SQL Relational Engine, via Reporting Services to SQL Analysis Services, etc).

Excel Services

To test identity delegation via Excel Services to the Relational Engine, first create a SQL Server view based on the SQL statement in the previous post.  Then create a new Excel Workbook, connect to SQL server and create a new PivotTable based on this view.  Expected results are shown in the screenshot below.

Testing identity delegation via Excel Services to Analysis Services is a little more involved.  First, create a simple cube in Analysis Services (or modify an existing one, even AdventureWorks - the makeup of the cube doesn’t matter at all).  Create some calculated measures based on the MDX query in the previous post.  The equivalent MDX for defining measures in a cube is (paste this after the CALCULATE statement):

CREATE MEMBER CurrentCube.Measures.User as UserName();
CREATE MEMBER CurrentCube.Measures.[CustomData] as CustomData()

Deploy the cube.  Now, re-open the Excel Workbook, connect to Analysis Services and base a second Pivot Table in the Excel Workbook on the cube (only use the two measures created above – ignore whatever else is in the cube).

Upload the workbook into a document library in SharePoint.  View it via Excel Services (after opening, you may need to select Data >> Refresh… to update the contents):

The highlighted cells (on the left show) that the current connection from Excel Services has been authenticated using Kerberos.  They also show that the user identity has been delegated through to the SQL Server Relational Engine correctly. 

The highlighted cell on the right shows that the user identity has been delegated to Analysis Services correctly.  Given there are multiple hops and multiple protocol transitions involved, it’s a reasonable conclusion that Kerberos is working.

Performance Point Services

To test identity delegation via Performance Point Services to Analysis Services requires creating a simple test dashboard in Dashboard Designer.  Use a connection configured with the ‘Per User Identity’ authentication setting.  Then create an analytical view either based on the cube created/modified above or simply use the MDX query defined in the previous post.

Deploy this to SharePoint and view:

The highlighted cells show that the user identity has been delegated through to Analysis Services correctly.

Reporting Services

By creating a report based on the SQL View described above, delegation via SSRS can be proven:

PowerPivot for SharePoint

PowerPivot for SharePoint doesn’t directly connect to the back end data source when a user is viewing a workbook containing PowerPivot data.  Therefore, no identity delegation tests are applicable.

Visio Services

To test delegation via Vision Services to the Relational Engine requires creating a web drawing and linking some shapes to external data (in this case, SQL Server).  First, create a simpler version of the above SQL view that returns information just about your connection:

select s.Session_Id, s.Login_Name, s.Host_name, c.Auth_Scheme,
case when c.Auth_Scheme = 'KERBEROS' then 1 else 0 end IsKerberos
from sys.dm_exec_connections c
inner join sys.dm_exec_sessions s
on s.session_id = c.session_id
where c.session_id = @@SPID

When run from Management Studio, this returns a single record:

Next, create a new Visio Diagram.  Link the shapes to this external data view:

Save this as a Web Drawing (*.vdw) into SharePoint, open it in Visio Services and click Refresh:

This shows that the user identity is successfully being delegated back to SQL Server.

Continued...

Continue reading in Part 4.