John Desch (Microsoft Consulting Services Data Insights)

Finding the Last Processed Date for Analysis Services Cubes, Measure Groups and Partitions

John Desch MSFT — Tue, 20 Sep 2016 02:42:52 GMT

Recently, a question was raised on one of the email aliases that I follow and I’ve seen the same basic question raised multiple times on various news groups. Initially, the question was “Where or how can I find the Last Processed Date for the cubes contained in an Analysis Services database?” As database sizes have increased, resulting in increased use of partitioning in Analysis Services cubes, that question has morphed to something closer to “Where or how can I find the Last Processed Date for the cubes and partitions contained in an Analysis Services database?” For the original question, the standard answer was to execute something like the following Dynamic Management View (DMV) query: SELECT * FROM $SYSTEM.MDSCHEMA_CUBES When executed in the context of the Adventure Works Multidimensional database, the following result is returned: When executed in the context of the tabular version of the Adventure Works database, the following (and interestingly very similar) result set is returned: If something a bit less verbose is desired, then a DMV query similar to the following is pretty nice: SELECT [CATALOG_NAME], CUBE_NAME, LAST_DATA_UPDATE FROM $System.MDSCHEMA_CUBES order by last_data_update DESC Executed against the multidimensional database, the following data set is returned. The same DMV query executed against the tabular version of Adventure Works returns the following partial data set. The DMV queries produce a result indicating the last date that the cube was processed, but there are a couple of problems with this approach. Regardless of whether the model is Multidimensional or Tabular, the MDSCHEMA_CUBES DMV returns data on when the various dimensions included in the cube. For example, we get information related to the Reseller dimension (CUBE_NAME = $Reseller), Promotion Dimension (CUBE_NAME = $Promotion), the Employee Dimension (CUBE_NAME=$Employee) and the Organization Dimension (CUBE_NAME=$Organization. It’s possible to eliminate the dimensions from the result by changing the DMV query to something similar to the following: SELECT [CATALOG_NAME], CUBE_NAME, LAST_DATA_UPDATE FROM $System.MDSCHEMA_CUBES where LEFT([CUBE_NAME],1) <>'$' order by last_data_update DESC Executed against the multidimensional version of the Adventure Works 2012 database, we get the following: When the same query is executed against the Tabular version of the Adventure Works database, the following result is returned: This surfaces another problem with querying the MDSCHEMA_CUBES DMV, since all tables in a tabular model are both dimensions and fact tables. For example, the Internet Sales table contains measures and is a partitioned table. In order to get any information regarding the Last Processed Date below the level of the cube, it’s really necessary to invoke the AMO API. To those unaccustomed to writing a bit of VB, C#, or PowerShell this may sound somewhat daunting but it’s actually pretty trivial to write the code to return the Last Processed Date for the Cube, Measure Groups, and Partitions. Writing in C#, the following code will work quite nicely to build out a console application in C#: using System; using System.Collections.Generic; using System.Linq; using System.Text; using System.Threading.Tasks; using Microsoft.AnalysisServices; namespace GetProcessedDate { class Program { static void Main(string[] args) { Server AsServer = new Server(); AsServer.Connect(string.Format("Data Source={0}",args[0].ToString())); Database AsDB = AsServer.Databases.FindByName(args[1].ToString().Trim()); foreach (Cube cub in AsDB.Cubes) { Console.WriteLine(string.Format("Cube Name = {0}, Last Processed Date = {1}", cub.Name.ToString(), cub.LastProcessed.ToString())); foreach (MeasureGroup mg in cub.MeasureGroups) { foreach (Partition part in mg.Partitions) { Console.WriteLine(string.Format("Measure Group = {0}, Partition = {1}, Last Processed Date = {2}", mg.Name.ToString(), part.ToString(), part.LastProcessed.ToString())); } } } AsServer.Disconnect(); AsServer.Dispose(); Console.ReadLine(); } } } It’s possible to do the same thing and obtain identical output using fewer lines of code with PowerShell, as in the example below: Param( [Parameter(Mandatory=$True)] [string]$ServerName, [Parameter(Mandatory=$True)] [string]$DataBaseName ) [System.Reflection.Assembly]::LoadWithPartialName("Microsoft.AnalysisServices") >$NULL $server = New-Object Microsoft.AnalysisServices.Server $server.connect($ServerName.ToString()) $database = $server.databases $asdb = $database[$DatabaseName.ToString()] foreach ($cube in $asdb.Cubes) { Write-Host "Cube Name = $($cube.name) , Last Processed Date = $($cube.LastProcessed)" foreach ($measuregroup in $cube.MeasureGroups) { foreach ($part in $measuregroup.Partitions) { Write-Host "MeasureGroup Name = $($measuregroup.name) , Partition Name = $($part.name) , Last Processed Date = $($part.LastProcessed)"}} } When executed against a multidimensional model, both return the following data: Cube Name = Adventure Works , Last Processed Date = 07/22/2016 16:15:47 MeasureGroup Name = Internet Sales , Partition Name = Internet_Sales_2005 , Last Processed Date = 07/22/2016 16:15:40 MeasureGroup Name = Internet Sales , Partition Name = Internet_Sales_2006 , Last Processed Date = 07/22/2016 16:15:41 MeasureGroup Name = Internet Sales , Partition Name = Internet_Sales_2007 , Last Processed Date = 07/22/2016 16:15:42 MeasureGroup Name = Internet Sales , Partition Name = Internet_Sales_2008 , Last Processed Date = 07/22/2016 16:15:42 MeasureGroup Name = Internet Orders , Partition Name = Internet_Orders_2005 , Last Processed Date = 07/22/2016 16:15:38 MeasureGroup Name = Internet Orders , Partition Name = Internet_Orders_2006 , Last Processed Date = 07/22/2016 16:15:42 MeasureGroup Name = Internet Orders , Partition Name = Internet_Orders_2007 , Last Processed Date = 07/22/2016 16:15:45 MeasureGroup Name = Internet Orders , Partition Name = Internet_Orders_2008 , Last Processed Date = 07/22/2016 16:15:42 MeasureGroup Name = Internet Customers , Partition Name = Customers_2005 , Last Processed Date = 07/22/2016 16:15:38 MeasureGroup Name = Internet Customers , Partition Name = Customers_2006 , Last Processed Date = 07/22/2016 16:15:41 MeasureGroup Name = Internet Customers , Partition Name = Customers_2007 , Last Processed Date = 07/22/2016 16:15:42 MeasureGroup Name = Internet Customers , Partition Name = Customers_2008 , Last Processed Date = 07/22/2016 16:15:44 MeasureGroup Name = Sales Reasons , Partition Name = Internet_Sales_Reasons , Last Processed Date = 07/22/2016 16:15:37 MeasureGroup Name = Reseller Sales , Partition Name = Reseller_Sales_2005 , Last Processed Date = 07/22/2016 16:15:45 MeasureGroup Name = Reseller Sales , Partition Name = Reseller_Sales_2006 , Last Processed Date = 07/22/2016 16:15:43 MeasureGroup Name = Reseller Sales , Partition Name = Reseller_Sales_2007 , Last Processed Date = 07/22/2016 16:15:44 MeasureGroup Name = Reseller Sales , Partition Name = Reseller_Sales_2008 , Last Processed Date = 07/22/2016 16:15:46 MeasureGroup Name = Reseller Orders , Partition Name = Reseller_Orders_2005 , Last Processed Date = 07/22/2016 16:15:46 MeasureGroup Name = Reseller Orders , Partition Name = Reseller_Orders_2006 , Last Processed Date = 07/22/2016 16:15:43 MeasureGroup Name = Reseller Orders , Partition Name = Reseller_Orders_2007 , Last Processed Date = 07/22/2016 16:15:45 MeasureGroup Name = Reseller Orders , Partition Name = Reseller_Orders_2008 , Last Processed Date = 07/22/2016 16:15:46 MeasureGroup Name = Sales Summary , Partition Name = Total_Sales_2005 , Last Processed Date = 07/22/2016 16:15:43 MeasureGroup Name = Sales Summary , Partition Name = Total_Sales_2006 , Last Processed Date = 07/22/2016 16:15:45 MeasureGroup Name = Sales Summary , Partition Name = Total_Sales_2007 , Last Processed Date = 07/22/2016 16:15:45 MeasureGroup Name = Sales Summary , Partition Name = Total_Sales_2008 , Last Processed Date = 07/22/2016 16:15:46 MeasureGroup Name = Sales Orders , Partition Name = Total_Orders_2005 , Last Processed Date = 07/22/2016 16:15:43 MeasureGroup Name = Sales Orders , Partition Name = Total_Orders_2006 , Last Processed Date = 07/22/2016 16:15:45 MeasureGroup Name = Sales Orders , Partition Name = Total_Orders_2007 , Last Processed Date = 07/22/2016 16:15:46 MeasureGroup Name = Sales Orders , Partition Name = Total_Orders_2008 , Last Processed Date = 07/22/2016 16:15:45 MeasureGroup Name = Sales Targets , Partition Name = Sales_Quotas , Last Processed Date = 07/22/2016 16:15:37 MeasureGroup Name = Financial Reporting , Partition Name = Finance , Last Processed Date = 07/22/2016 16:15:40 MeasureGroup Name = Exchange Rates , Partition Name = Currency_Rates , Last Processed Date = 07/22/2016 16:15:39 Cube Name = Mined Customers , Last Processed Date = 07/22/2016 16:15:48 When executed with the Tabular version of the Adventure Works database, both return the following data set. Cube Name = Model , Last Processed Date = 04/03/2015 11:11:37 MeasureGroup Name = Currency , Partition Name = Currency , Last Processed Date = 04/03/2015 11:11:19 MeasureGroup Name = Customer , Partition Name = Customer , Last Processed Date = 04/03/2015 11:11:19 MeasureGroup Name = Date , Partition Name = Date , Last Processed Date = 04/03/2015 11:11:20 MeasureGroup Name = Employee , Partition Name = Employee , Last Processed Date = 04/03/2015 11:11:20 MeasureGroup Name = Geography , Partition Name = Geography , Last Processed Date = 04/03/2015 11:11:18 MeasureGroup Name = Product , Partition Name = Product , Last Processed Date = 04/03/2015 11:11:19 MeasureGroup Name = Product Category , Partition Name = Product Category , Last Processed Date = 04/03/2015 11:11:19 MeasureGroup Name = Product Subcategory , Partition Name = Product Subcategory , Last Processed Date = 04/03/2015 11:11:19 MeasureGroup Name = Promotion , Partition Name = Promotion , Last Processed Date = 04/03/2015 11:11:19 MeasureGroup Name = Reseller , Partition Name = Reseller , Last Processed Date = 04/03/2015 11:11:17 MeasureGroup Name = Sales Territory , Partition Name = Sales Territory , Last Processed Date = 04/03/2015 11:11:20 MeasureGroup Name = Internet Sales , Partition Name = Internet Sales 2005 , Last Processed Date = 09/14/2016 17:29:51 MeasureGroup Name = Internet Sales , Partition Name = Internet Sales 2006 , Last Processed Date = 09/14/2016 17:29:54 MeasureGroup Name = Internet Sales , Partition Name = Internet Sales 2007 , Last Processed Date = 09/14/2016 17:29:58 MeasureGroup Name = Internet Sales , Partition Name = Internet Sales 2008 , Last Processed Date = 09/14/2016 17:30:03 MeasureGroup Name = Internet Sales , Partition Name = Internet Sales 2009 , Last Processed Date = 09/14/2016 17:30:06 MeasureGroup Name = Internet Sales , Partition Name = Internet Sales 2010 , Last Processed Date = 09/14/2016 17:30:09 MeasureGroup Name = Product Inventory , Partition Name = Product Inventory 2005 , Last Processed Date = 04/03/2015 11:11:18 MeasureGroup Name = Product Inventory , Partition Name = Product Inventory 2006 , Last Processed Date = 04/03/2015 11:11:21 MeasureGroup Name = Product Inventory , Partition Name = Product Inventory 2007 , Last Processed Date = 04/03/2015 11:11:25 MeasureGroup Name = Product Inventory , Partition Name = Product Inventory 2008 , Last Processed Date = 04/03/2015 11:11:28 MeasureGroup Name = Product Inventory , Partition Name = Product Inventory 2009 , Last Processed Date = 04/03/2015 11:11:31 MeasureGroup Name = Product Inventory , Partition Name = Product Inventory 2010 , Last Processed Date = 04/03/2015 11:11:34 MeasureGroup Name = Reseller Sales , Partition Name = Reseller Sales 2005 , Last Processed Date = 04/03/2015 11:11:18 MeasureGroup Name = Reseller Sales , Partition Name = Reseller Sales 2006 , Last Processed Date = 04/03/2015 11:11:22 MeasureGroup Name = Reseller Sales , Partition Name = Reseller Sales 2007 , Last Processed Date = 04/03/2015 11:11:24 MeasureGroup Name = Reseller Sales , Partition Name = Reseller Sales 2008 , Last Processed Date = 04/03/2015 11:11:27 MeasureGroup Name = Reseller Sales , Partition Name = Reseller Sales 2009 , Last Processed Date = 04/03/2015 11:11:30 MeasureGroup Name = Reseller Sales , Partition Name = Reseller Sales 2010 , Last Processed Date = 04/03/2015 11:11:33 MeasureGroup Name = Sales Quota , Partition Name = Sales Quota , Last Processed Date = 04/03/2015 11:11:17

Using SQL Server Stored Procedures to Dynamically Drop and Recreate Indexes

John Desch MSFT — Sat, 30 Jul 2016 04:36:35 GMT

Recently, I've been working on a project that of necessity involves periodically updating data in some reasonably large tables that exist in an Operational Data Store (ODS). This particular ODS is used for both reporting via SQL Server Reporting Services and staging data for use in a SQL Server Analysis Services database. Since a number of the tables in the ODS are used for reporting purposes, it's not entirely surprising that the report designers have created a few indexes to help report performance. I don't have a problem with indexes, but as any experienced DBA is well aware, the more and larger the indexes the greater the impact on the performance of inserts and updates. The magnitude of the performance impact was really brought home when a simple update on a 12 million row table that normally completed in roughly three minutes had to be killed at the two hour mark. On further investigation, it was found that over 30 indexes had been added to the table in question. So to address the immediate problem and allow the update to complete in a reasonable time period, DROP INDEX and CREATE INDEX commands were scripted out and added to a stored procedure which would first drop the indexes then run the update statement and finally recreate the indexes. That worked well for a couple of days and then performance again began to degrade. When this episode of performance degradation was investigated, it was found that of the indexes that had been scripted out and added to the stored procedure only one remained and several additional indexes had been created. Not wishing to revise a rather lengthy stored proc on an almost daily basis, after a brief bit of research, I found a blog posting by Percy Reyes entitled Script out all SQL Server Indexes in a Database using T-SQL. That was great, but only covered NONCLUSTERED indexes and since we were seeing both CLUSTERED and NONCLUSTERED indexes, it would need a bit of revision. Coincidentally, at about the same time there was some serious talk of adding COLUMNSTORE indexes on one or two of these tables, which would essentially cause any update statement to fail. The possibility of having to contend with COLUMNSTORE indexes in addition to CLUSTERED and NONCLUSTERED indexes would necessitate a reasonably significant revision to the T-SQL presented in Percy's blog, especially since it would be necessary to dynamically drop and then recreate indexes. With those bits of information, it was time to formulate a plan, which would mean accomplishing the following:

Capturing and storing the names of tables, their associated indexes and the definitions of those indexes
Dropping the indexes after the definitions had been safely stored
Recreating the indexes from stored definitions using correct syntax

A relatively simple three step task, the first of which was to create a stored proc that would capture the names of the tables, associated indexes and the definitions of those indexes. That lead to creation of the following SQL Server stored procedure: CREATE PROCEDURE [dbo].[sp_GetIndexDefinitions] as IF NOT EXISTS (SELECT SCHEMA_NAME FROM INFORMATION_SCHEMA.SCHEMATA WHERE SCHEMA_NAME='WORK') BEGIN EXEC sp_executesql N'CREATE SCHEMA WORK' END IF OBJECT_ID('[WORK].[IDXDEF]','U') IS NOT NULL DROP TABLE [WORK].[IDXDEF] CREATE TABLE WORK.IDXDEF (SchemaName NVARCHAR(100), TableName NVARCHAR(256), IndexName NVARCHAR(256), IndexDef NVARCHAR(max)) DECLARE @SchemaName VARCHAR(100) DECLARE @TableName VARCHAR(256) DECLARE @IndexName VARCHAR(256) DECLARE @ColumnName VARCHAR(100) DECLARE @is_unique VARCHAR(100) DECLARE @IndexTypeDesc VARCHAR(100) DECLARE @FileGroupName VARCHAR(100) DECLARE @is_disabled VARCHAR(100) DECLARE @IndexOptions VARCHAR(max) DECLARE @IndexColumnId INT DECLARE @IsDescendingKey INT DECLARE @IsIncludedColumn INT DECLARE @TSQLScripCreationIndex VARCHAR(max) DECLARE @TSQLScripDisableIndex VARCHAR(max) DECLARE CursorIndex CURSOR FOR SELECT schema_name(st.schema_id) [schema_name], st.name, si.name, CASE WHEN si.is_unique = 1 THEN 'UNIQUE ' ELSE '' END , si.type_desc, CASE WHEN si.is_padded=1 THEN 'PAD_INDEX = ON, ' ELSE 'PAD_INDEX = OFF, ' END + CASE WHEN si.allow_page_locks=1 THEN 'ALLOW_PAGE_LOCKS = ON, ' ELSE 'ALLOW_PAGE_LOCKS = OFF, ' END + CASE WHEN si.allow_row_locks=1 THEN 'ALLOW_ROW_LOCKS = ON, ' ELSE 'ALLOW_ROW_LOCKS = OFF, ' END + CASE WHEN INDEXPROPERTY(st.object_id, si.name, 'IsStatistics') = 1 THEN 'STATISTICS_NORECOMPUTE = ON, ' ELSE 'STATISTICS_NORECOMPUTE = OFF, ' END + CASE WHEN si.ignore_dup_key=1 THEN 'IGNORE_DUP_KEY = ON, ' ELSE 'IGNORE_DUP_KEY = OFF, ' END + 'SORT_IN_TEMPDB = OFF' + CASE WHEN si.fill_factor>0 THEN ', FILLFACTOR =' + cast(si.fill_factor as VARCHAR(3)) ELSE '' END AS IndexOptions ,si.is_disabled , FILEGROUP_NAME(si.data_space_id) FileGroupName FROM sys.tables st INNER JOIN sys.indexes si on st.object_id=si.object_id WHERE si.type>0 and si.is_primary_key=0 and si.is_unique_constraINT=0 --and schema_name(tb.schema_id)= @SchemaName and tb.name=@TableName and st.is_ms_shipped=0 and st.name<>'sysdiagrams' ORDER BY schema_name(st.schema_id), st.name, si.name open CursorIndex FETCH NEXT FROM CursorIndex INTO @SchemaName, @TableName, @IndexName, @is_unique, @IndexTypeDesc, @IndexOptions, @is_disabled, @FileGroupName WHILE (@@fetch_status=0) BEGIN DECLARE @IndexColumns VARCHAR(max) DECLARE @IncludedColumns VARCHAR(max) SET @IndexColumns='' SET @IncludedColumns='' DECLARE CursorIndexColumn CURSOR FOR SELECT col.name, sic.is_descending_key, sic.is_included_column FROM sys.tables tb INNER JOIN sys.indexes si on tb.object_id=si.object_id INNER JOIN sys.index_columns sic on si.object_id=sic.object_id and si.index_id= sic.index_id INNER JOIN sys.columns col on sic.object_id =col.object_id and sic.column_id=col.column_id WHERE si.type>0 and (si.is_primary_key=0 or si.is_unique_constraINT=0) and schema_name(tb.schema_id)=@SchemaName and tb.name=@TableName and si.name=@IndexName ORDER BY sic.index_column_id OPEN CursorIndexColumn FETCH NEXT FROM CursorIndexColumn INTO @ColumnName, @IsDescendingKey, @IsIncludedColumn WHILE (@@fetch_status=0) BEGIN IF @IsIncludedColumn=0 SET @IndexColumns=@IndexColumns + @ColumnName + CASE WHEN @IsDescendingKey=1 THEN ' DESC, ' ELSE ' ASC, ' END ELSE SET @IncludedColumns=@IncludedColumns + @ColumnName +', ' FETCH NEXT FROM CursorIndexColumn INTO @ColumnName, @IsDescendingKey, @IsIncludedColumn END CLOSE CursorIndexColumn DEALLOCATE CursorIndexColumn SET @IndexColumns = substring(@IndexColumns, 0, len(ltrim(rtrim(@IndexColumns)))) SET @IncludedColumns = CASE WHEN len(@IncludedColumns) >0 THEN substring(@IncludedColumns, 0, len(@IncludedColumns)) ELSE '' END SET @TSQLScripCreationIndex ='' SET @TSQLScripDisableIndex ='' SET @TSQLScripCreationIndex='CREATE '+ @is_unique + @IndexTypeDesc + ' INDEX ' +QUOTENAME(@IndexName)+' ON ' +QUOTENAME(@SchemaName) +'.'+ QUOTENAME(@TableName)+ CASE WHEN @IndexTypeDesc = 'NONCLUSTERED COLUMNSTORE' THEN ' ('+@IncludedColumns+') ' WHEN @IndexTypeDesc = 'CLUSTERED COLUMNSTORE' THEN ' ' ELSE ' ('+@IndexColumns+') ' END + CASE WHEN @IndexTypeDesc = 'NONCLUSTERED COLUMNSTORE' and len(@IncludedColumns)>0 THEN '' when @IndexTypeDesc = 'CLUSTERED COLUMNSTORE' THEN '' ELSE CASE WHEN LEN(@IncludedColumns)>0 THEN CHAR(13) +'INCLUDE (' + @IncludedColumns+ ')' ELSE '' END END + CASE WHEN @IndexTypeDesc not like ('%COLUMNSTORE%') THEN CHAR(13) + 'WITH (' + @IndexOptions + ') ' + ' ON ' + QUOTENAME(@FileGroupName) ELSE '' END + ';' INSERT INTO [WORK].[IDXDEF] (Schemaname,TableName,IndexName,IndexDef) values (@SchemaName, @TableName, @IndexName, @TSQLScripCreationIndex) FETCH NEXT FROM CursorIndex INTO @SchemaName, @TableName, @IndexName, @is_unique, @IndexTypeDesc, @IndexOptions, @is_disabled, @FileGroupName END CLOSE CursorIndex DEALLOCATE CursorIndex When that tested out, it was time for the next step of the task and simply dynamically dropping the indexes. But I wanted to ensure that when the indexes were dropped, that the index definitions would be safely stored (I'm like most other DBAs and sort of enjoy being employed). That resulted in creation of the following stored proc: CREATE PROCEDURE [dbo].[sp_DropIndexes] as EXEC sp_GetIndexDefinitions DECLARE @DropIndex NVARCHAR(max) DECLARE @SchemaName NVARCHAR(256) DECLARE @TableName NVARCHAR(256) DECLARE @IndexName NVARCHAR(256) DECLARE CursorIDXDrop CURSOR FOR SELECT DISTINCT ss.name AS schemaname, st.name AS tblname, si.name AS indexnname FROM sys.tables st INNER JOIN sys.schemas ss ON st.schema_id=ss.schema_id INNER JOIN sys.indexes si ON st.object_id=si.object_id WHERE si.type<>0 AND st.is_ms_shipped=0 AND st.name<>'sysdiagrams' AND (is_primary_key=0 AND is_unique_constraint=0) OPEN CursorIDXDrop FETCH NEXT FROM CursorIDXDrop INTO @SchemaName, @TableName, @IndexName WHILE @@FETCH_STATUS=0 BEGIN SET @DropIndex= 'DROP INDEX ' + QUOTENAME(@IndexName) + ' ON ' + QUOTENAME(@SchemaName) + '.' + QUOTENAME(@TableName) EXEC sp_executesql @DropIndex FETCH NEXT FROM CursorIDXDrop INTO @SchemaName, @TableName, @IndexName END CLOSE CursorIDXDrop DEALLOCATE CursorIDXDrop After that worked, with the index definitions safely stored so that I could manually re-create them if necessary, it was time to move on to the third step of dynamically re-creating the indexes. That resulted in creation of the following stored proc: CREATE PROCEDURE [dbo].[sp_RebuildIndexes] as IF OBJECT_ID('[WORK].[IDXDEF]','U') IS NOT NULL BEGIN DECLARE @Command nvarchar(max) DECLARE @IndexCmd nvarchar(max) DECLARE @NumRows int DECLARE CursorIndexes CURSOR for SELECT DISTINCT CAST(IndexDef as nvarchar(max)) as IndexDef from work.IDXDEF SET @NumRows=0 OPEN CursorIndexes FETCH NEXT FROM CursorIndexes into @Command WHILE @@FETCH_STATUS=0 BEGIN EXEC sp_executesql @Command FETCH NEXT FROM CursorIndexes into @Command SET @NumRows=@NumRows+1 END; CLOSE CursorIndexes DEALLOCATE CursorIndexes --PRINT LTRIM(RTRIM(CAST(@NumRows as varchar(10)))) + ' Indexes Recreated from stored definitions' END By having my update script call the sp_DropIndexes before the update and then call the sp_RebuildIndexes, it was very easy to drop the indexes, run updates and then re-create the indexes in a reasonable time period without the necessity of having to continually revise code.

Scaling Out with Analysis Services

John Desch MSFT — Mon, 17 Aug 2015 16:09:00 GMT

Scaling Out with Analysis Services

A few months ago, I had the opportunity to work with a customer who had been intermittently encountering a variety of errors when attempting to connect to an Analysis Services server. The production environment consisted of a dedicated processing server and a dedicated query server. Both of the servers were configured with eight (8) logical cores and 64GB of RAM and were running SQL Server 2008 R2 Analysis Server.

Owing to a business requirement that the data be regularly refreshed at 15 minute intervals, the processing server was used to execute a full process of a single database every 15 minutes. Since the query server was not used for processing, the database was being refreshed using the Synchronization feature that is native to Analysis Services. All-in-all, this is a recognized practice for scaling out Analysis Services that was described by Denny Lee and Nicholas Dritsas in 2007 in their whitepaper entitled SQL Server Best Practices Article.

Shortly after the scale-out solution was implemented, the customer began experiencing periodic issues with connectivity. Initially, the connectivity issues were very sporadic and while there was some concern the issues weren’t considered to be critical. Unfortunately, the issues began to occur with increasing frequency over a period of several months and eventually reached a point that upper management considered it critical that the issues be addressed. At this point, a Profiler Trace on the Analysis Services service was captured and the analysis yielded some interesting (if not somewhat confusing) results. The particular trace was captured over a period of 127 minutes so data covering multiple synchronization cycles and a total of 7539 MDX queries was collected. That trace included information on 291 connectivity errors over the span of 127 minutes. The connectivity failures included messages similar to the following:

A connection cannot be made. Ensure that the server is running
A connection attempt failed because the connected party did not properly respond after a period of time.
Either the user, **\********, does not have access to the ****** database, or the database does not exist….
Server: The operation was cancelled because of locking conflicts. The 'Employee' attribute in the 'Security' dimension has a generated dimension security expression that is not valid. Query Text: SELECT {} on 0, [Reporting Date].[Year - Quarter - Month - Week - Date].[Date] on 1 from ( SELECT STRTOSET(@Week) on 0 FROM ( SELECT STRTOSET(@QuarterMonth) on 0 FROM ( SELECT STRTOSET (@Year) on 0 FROM [cube] ) ) )
Profiler traces indicating that users were attempting to connect using the NT AUTHORITY\ANONYMOUS LOGON account

Given the variety of errors, a number of theories rapidly evolved regarding the root cause or causes. Those theories included network issues, the expression used to implement dimension security, locking conflicts, and server load. Since there were a number of competing theories as to the root cause, it was decided to investigate each systematically.

Setting up the Test Environment:

In order to test out the various theories that had been advanced, it was decided to create a test environment that consisted of three servers:

One Windows 2008 R2 server running Analysis Services 2012 to serve as a “Source” server for synchronizations.
One Windows 2008 R2 server running Analysis Services 2012 to serve as a “Target” server for synchronizations.
One Windows 2012 Server running a custom multi-threaded application capable of executing up to 100 MDX queries in parallel.

The strategy, at this point, was very simply to execute an Analysis Services Synchronization while submitting a total of 5000 MDX queries via the multi-threaded application. To monitor the behavior of the target server, a Profiler trace and Performance Monitor data collector were used.

Network Issue Theory:

There were a number of SQL Server Profiler traces captured, which hinted at the possibility of network issues due to a number of permissions errors (i.e. “Either the user, NA\someuser, does not have access to the or…”) , and several “NT AUTHORITY\ANONYMOUS LOGON” entries. Coincidentally, all of these errors were logged in the trace while a Synchronization operation was being executed. One of the more interesting profiler traces indicated 291 permissions related errors during execution of 7539 queries in the span of 127 minutes while a Synchronization transaction was running. Unfortunately, for one reason or other, the customer never captured a Network Monitor or WireShark network sniff to really make a determination regarding this theory, so it couldn’t be categorically ruled out.

Security Expression Theory:

To test this theory, using an analog database with very similar characteristics, security expressions modeled on the MDX expressions used in the production database, and a library of 5000 MDX queries modeled on those captured in multiple profiler traces, the UCASE VBA function was first removed from the Dimension Security expression and executed a total of 5000 MDX queries from two client applications, with each running 100 queries in parallel. That resulted in errors due to Lock Conflicts, permissions errors (i.e. “Either the user, NA\someuser, does not have access to the or…”) , and several “NT AUTHORITY\ANONYMOUS LOGON” entries while the Synchronization operation was being executed. To test further, the UCASE VBA function was removed from the DefaultMember expression and the same test conditions repeated, resulting in errors due to Locking Conflicts, permissions errors, and the appearance of “NT AUTHORITY\ANONYMOUS LOGON” entries in the profiler trace. Finally, the security expression was completely removed from the secured dimension and the test was repeated which still resulted in errors.

(Note: At this point, I’m definitely seeing a pattern here that involves queries executing while the Synchronization operation is executing.)

Server Load Theory:

In one of the profiler traces, it had been noted that there were a total of 530 concurrent connections to the server. Since the server had eight (8) cores, and running Analysis Services 2012, the maximum number of threads in the query thread pool would have been throttled at 16, so queuing in the query thread pool was not at all unexpected and raised questions about load. To test that theory, 5000 MDX queries were submitted to the test query server via 200 concurrent connections (two instances of the multi-threaded application) while a Synchronization operation was executing. With that load, the entire range of connectivity errors being reported could be readily reproduced. It was noted that, even under peak load, there was nothing in the PerfMon counter traces to indicate that there were issues with disk performance, given that all of the Logical and Physical disk counters normally considered for performance (i.e. Avg. Disk sec/Read, Avg. Disk sec/Write, and Avg. Disk sec/Transfer) were all under a value of 0.10. Gradually the number of concurrent connections was reduced to 16, then 14, then 8, then 4, and finally to 1, with synchronization jobs running periodically during the test runs. An analysis of those data yielded evidence that there was a very definite relationship between queuing in both the Query Thread pool and the Short Parsing Thread pool with the connectivity errors. By increasing the number of concurrent connections and queries executed in parallel the connectivity errors could be readily reproduced.

What was actually happening:

Armed with some pretty clean data from a controlled environment and deep knowledge of how Synchronization works in Analysis Services, it was reasonably clear that the connectivity issues were directly related to the synchronization transaction. Essentially, what happens when a synchronization between servers is executed, is that the server compares the timestamps and versions of the files on the disk of the target server with timestamps and version of the files on the source server. Files on the target that are outdated are updated to the newer timestamp and version with the files from the source server, however, the older files remain on disk until the transaction is committed. Analysis Services uses a two phase commit process, so it first acquires a read commit lock in phase 1 of the commit. In phase 2 of the commit, a server level lock is acquired on the Master Version Map (master.vmp), while the ObjectID values for the new files are written to disk and the older files are deleted. If there are queries or other operations executing when the server level lock is acquired, the server will wait for the period of time specified in the ForceCommitTimeout (default is 30 seconds) and then terminate any executing queries or commands, which results in the Lock Conflict error. Until the server level lock is released and the updated Master Version Map is closed, access to objects on the server is denied, resulting in the permissions errors. If Phase 2 of the commit executes for an extended period of time, which would be the case with a reasonably large database on a server under load, the permissions errors would occur for an extended period of time because a read lock could not be obtained to allow execution of other commands and queries.

A different conceptual approach:

Armed with a much better understanding of what was actually happening, it was reasonably clear that eliminating the connectivity issue would require an approach that did not involve use of the Synchronization functionality native to Analysis Services. After reading Scale-Out Querying for Analysis Services with Read-Only Databases, by Denny Lee and Kay Unkroth, the idea of Database Copying seemed the obvious solution. Just one or two little problems with using either a SAN Snapshot or Robocopy. The SAN in the production environment wasn’t configured to support snapshots, so that option wasn’t viable. That left Robocopy, which, like a SAN Snapshot, would have required taking the Query server offline long enough to detach the database, copy files from source to target and then attach the refreshed database. Clearly time to think a little bit outside the box and build on the Scale-out concept.

Having a pair of load balanced query servers serving data from a single shared read-only database would provide a solution with no downtime. That reduced the problem to how can the data be refreshed on the query servers given the limitations of the production environment. Since iSCSI can be used to implement Pooled Storage, it is possible to attach the same LUN to multiple machines. Extending that concept, why not use iSCSI to attach a pair of LUNs to three machines (one dedicated to processing and two dedicated to query execution). On the processing server, keep one of the two LUNs Online in read-write mode and on the two query servers keep one of the two LUNs Online in read-only mode. After a processing cycle,

detach the freshly processed database from the processing server
mark the LUN with the freshly processed data as Offline
stop query server 1
allow the existing connections to drain off
detach the read-only database
mark the LUN containing stale data as Offline
mark the LUN with the newly refreshed data as Online and read-only
attach the freshly processed database in read-only mode
bring query server 1 back online
stop query server 2
allow the existing connections to drain off
detach the read-only database
mark the LUN containing stale data as Offline
mark the LUN with the newly refreshed data as Online and read-only
attach the freshly processed database in read-only mode
bring query server 2 back online
on the processing server mark the LUN with the stale data as Online and set it to read-write
attach the database in read-write mode

Setting up the Proof of Concept Environment:

In order to test out the various theories that had been advanced, it was decided to create a test environment that consisted of a total of five servers:

1 Domain Controller
1 Analysis Services server for dedicated processing
1 Windows 2008 R2 Server running Network Load Balancing (NLB)
2 Analysis Services 2012 servers that were configured as part of the NLB cluster.

iSCSI was used to mount the same two LUNs on the three Analysis Services servers. On the processing server, one LUN was marked as “Online” and “ReadWrite” with the other marked “Offline”. On the two Query servers, the LUN that was “Offline” on the processing server as marked as “Online” and “ReadOnly” while the LUN that was “Online” on the processing server was marked “Offline”.

A new NLB cluster was created and assigned a static IP address. The two query servers were then added to the NLB cluster. After that, an “NLB” DNS entry was created on the DC referencing the IP address assigned to the NLB cluster. Service Principal Names (SPNs) were also created for the NLB cluster as well as the two Analysis Services servers that were serving as query servers.

The next major task involved creation of code to handle detaching the database on the processing server as well as the query servers. To accomplish that, the following C# code was written to provide a simple command line application, which could be used on all three of the servers in the POC.

using System;

using System.Collections.Generic;

using System.Linq;

using System.Text;

using System.Threading.Tasks;

using Microsoft.AnalysisServices;

namespace Detach

{

class Program

{

static void Main(string[] args)

{

Server asServer = new Server();

asServer.Connect("Localhost");

Database asDB = asServer.Databases.FindByName("ChangingDims");

try

{

asDB.Detach();

}

catch (Exception goof)

{

Console.WriteLine("Database is not online");

}

finally

{

asServer.Disconnect();

asServer.Dispose();

}

Console.WriteLine("Take Drive S: offline then bring the other LUN online");

Console.ReadLine();

}

One of the goals of the POC was to eliminate connectivity issues and simply taking one of the two query servers offline would have resulted in a number of connectivity incidents (and probably a relatively large number of calls to the DBA team because users were suddenly disconnected). This was primary rationale for implementing a Network Load Balancing (NLB) cluster, since using the DrainStop command would allow one server to be taken offline without interrupting existing connections while the other server in the cluster handled any new connection requests. In a production environment, it would have been highly desirable to automate the process using either PowerShell (see Symon Perriman’s Blog post) or C# code, however, for the POC the Network Load Balancing Manager interface on the NLB cluster server was used. This entailed selecting the host from the cluster then implementing a “DrainStop”.

A second small C# application would be used on the Query Servers to monitor the number of connections that were actively executing queries after a server had been placed in a “DrainStop” status. Once the active connections had been allowed to “bleed off”, the application would then detach the database. In order to accomplish that, it was necessary to periodically check the number of connections and set a low limit threshold for the number of connections since the Monitor Application and Profiler each had one connection to the server while the Stress Application had two open connections to the server. It probably should be noted that Profiler traces were being captured on both of the query servers using local instances of Profiler. That code is shown below:

using System;

using System.Collections.Generic;

using System.Linq;

using System.Text;

using System.Data;

using Microsoft.AnalysisServices;

using Microsoft.AnalysisServices.AdomdClient;

using System.Timers;

namespace Query_Detach

{

class Program

{

static void Main(string[] args)

{

int tstart = (int)DateTime.Now.TimeOfDay.TotalSeconds;

int tcurrent;

AdomdConnection AdomdConn = new AdomdConnection();

AdomdConn.ConnectionString = "Data Source=localhost";

AdomdConn.Open();

DataSet ds = new DataSet();

while (true)

{

tcurrent = (int)DateTime.Now.TimeOfDay.TotalSeconds;

if ((tcurrent-30)>tstart)

{

Console.WriteLine(string.Format("Kicked at {0}", DateTime.Now.Second.ToString()));

Console.WriteLine(tcurrent.ToString() + " : " + tstart.ToString());

tstart=tcurrent;

ds.Clear();

ds=AdomdConn.GetSchemaDataSet(AdomdSchemaGuid.Connections, null);

Console.WriteLine(ds.Tables[0].Rows.Count.ToString());

/* The value 4 was used below to account for the following connections:

* FlightRecorder

* Profiler Trace

* Two connections for the Stress application

* The Query Monitor and Database Detach application

if (ds.Tables[0].Rows.Count <= 4)

{

break;

}

else

{

Console.WriteLine(string.Format("{0} connections are currently active",ds.Tables[0].Rows.Count.ToString()));

}

AdomdConn.Close();

AdomdConn.Dispose();

Server asServer = new Server();

asServer.Connect("localhost");

Database asDB = asServer.Databases.FindByName("ChangingDims");

try

{

asDB.Detach();

}

catch (AmoException goof)

{

Console.WriteLine("Database is not online");

}

finally

{

asServer.Disconnect();

asServer.Dispose();

}

Console.WriteLine("Take Drive S: offline then bring the other LUN online");

Console.ReadLine();

}

When the application executed, the screen displayed the following message:

Take Drive S: offline then bring the other LUN online

If this was going to be a production environment, additional code would have been added to handle swapping out the drives but since this was a POC, changing the status of the two LUNs on the servers was handled using Server Manager as illustrated below. When marking a drive as “Offline”, Server Manager was opened, then Disk Management was selected and then the drive status was changed. Server Manager was also used to bring the other LUN to an “Online” state. Once the LUN containing the freshly processed database had been marked as online, it was necessary to set the LUN to ReadOnly status on the targeted query server. Again, a relatively simple C# application was created to provide a command line utility using the following code:

using System;

using System.Collections.Generic;

using System.Linq;

using System.Text;

using System.Threading.Tasks;

using System.IO;

using Microsoft.AnalysisServices;

namespace Attach

{

class Program

{

static void Main(string[] args)

{

Helper setDrives = new Helper();

setDrives.ExecuteCommand("diskpart -s c:\\ctrl\\setdrives.txt");

Server asServer = new Server();

asServer.Connect("localhost");

try

{

asServer.Attach("S:\\Data\\CHANGINDDIMS.0.db", ReadWriteMode.ReadOnly);

Console.WriteLine("Database Attached");

}

catch (AmoException goof)

{

Console.WriteLine(goof.Message.ToString());

}

finally

{

asServer.Disconnect();

asServer.Dispose();

}

Console.ReadLine();

}

class Helper

{

public void ExecuteCommand(object Command)

{

try

{

System.Diagnostics.ProcessStartInfo procStart = new System.Diagnostics.ProcessStartInfo("cmd", "/c" + Command);

procStart.RedirectStandardOutput = true;

procStart.UseShellExecute = false;

procStart.CreateNoWindow = true;

System.Diagnostics.Process proc = new System.Diagnostics.Process();

proc.StartInfo = procStart;

proc.Start();

string result = proc.StandardOutput.ReadToEnd();

System.Diagnostics.Trace.WriteLine(result.ToString());

}

catch (Exception goof)

{

System.Diagnostics.Trace.WriteLine(goof.Message.ToString());

}

The text of the “setdrives.txt” file was:

Select volume 3

assign letter=S

attrib volume set Readonly

Upon successful attachment of the database in Read-Only mode, the following message was displayed on screen:

Database Attached

At this point, the Network Load Balancing Manager interface on the NLB cluster server was used to bring the server back online by selecting the server and then the Resume option. After the process had been completed with one query server, the same process was applied to the second query server in the cluster.

Once both query servers were back online, it was time to prepare the dedicated processing server for the next processing operation, which meant bringing the LUN containing the “stale” database online in ReadWrite mode and then attaching the database in Read-Write mode. To handle the tasks of , it was decided to write a small C# command line application to assign the drive letter S to the drive containing stale data then clear the ReadOnly attribute to make the drive “ReadWrite.” After the drive had been configured, the application would then attach the stale database so that the server would be ready for the next processing cycle. That code is below.

using System;

using System.Collections.Generic;

using System.Linq;

using System.Text;

using System.Threading.Tasks;

using System.IO;

using Microsoft.AnalysisServices;

namespace Proc_Attach

{

class Program

{

static void Main(string[] args)

{

Helper setDrives = new Helper();

setDrives.ExecuteCommand("diskpart -s c:\\ctrl\\setdrives.txt");

Server asServer = new Server();

asServer.Connect("localhost");

try

{

asServer.Attach("S:\\Data\\CHANGINDDIMS.0.db", ReadWriteMode.ReadWrite);

Console.WriteLine("Database Attached");

}

catch (AmoException goof)

{

Console.WriteLine(goof.Message.ToString());

}

finally

{

asServer.Disconnect();

asServer.Dispose();

}

Console.ReadLine();

}

class Helper

{

public void ExecuteCommand(object Command)

{

try

{

System.Diagnostics.ProcessStartInfo procStart = new System.Diagnostics.ProcessStartInfo("cmd", "/c" + Command);

procStart.RedirectStandardOutput = true;

procStart.UseShellExecute = false;

procStart.CreateNoWindow = true;

System.Diagnostics.Process proc = new System.Diagnostics.Process();

proc.StartInfo = procStart;

proc.Start();

string result = proc.StandardOutput.ReadToEnd();

System.Diagnostics.Trace.WriteLine(result.ToString());

}

catch (Exception goof)

{

System.Diagnostics.Trace.WriteLine(goof.Message.ToString());

}

The setdrives.txt file referenced in the application code contained the following lines:

Select volume 3

assign letter=S

attrib volume Clear Readonly

Testing it out:

After spending a bit of time to get the servers configured and write a bit of code, it was time to see if this approach would address the connectivity issues that the customer had been experiencing. Since one profiler trace included information on 291 connectivity errors during execution of 7539 MDX queries over eight (8) processing cycles, there was a basis for making some comparisons. Not being one who is particularly afraid of pushing boundaries, it was decided to run a larger number of queries using a stress tool over 6 processing cycles. Each processing cycle included the following steps:

Process the database
Detach the freshly processed database from the processing server
Mark the LUN with the freshly processed data as Offline
DrainStop query server 1
Allow the existing connections to drain off
Detach the read-only database
Mark the LUN containing stale data as Offline
Mark the LUN with the newly refreshed data as Online and read-only
Attach the freshly processed database in read-only mode
Bring query server 1 back online by Resuming from NLB
DrainStop query server 2
Allow the existing connections to drain off
Detach the read-only database
Mark the LUN containing stale data as Offline
Mark the LUN with the newly refreshed data as Online and read-only
Attach the freshly processed database in read-only mode
Bring query server 2 back online by Resuming from NLB
Mark the LUN with the stale data on the Processing Server as Online and set it to read-write
Attach the database in read-write mode
Begin the next processing cycle

Using the approach outlined here, a total of 21540 MDX queries were executed across 6 processing cycles with no errors.

Notes on Deciding whether to implement a Tabular or Multidimensional Model

John Desch MSFT — Mon, 27 Jul 2015 15:41:00 GMT

Within the Business Intelligence community, when the discussion comes to Microsoft SQL Server Analysis Services, there are a couple
of questions that typically arise. One of those questions is whether it’s possible to easily (read that as programmatically) convert from one model to
another. Typically, the other is whether to implement a solution that uses a Multidimensional model or a solution that is based on a Tabular model.

The answer to the first question is easy and relatively straight forward. Switching from one model to another essentially requires
starting from scratch. I know it’s not necessarily a pleasant thought, but there are a number of significant differences between the xVelocity engine used for
Tabular models and the OLAP engine that is used for Multidimensional models that make programmatic conversions, at least as of this writing, untenable. Consequently,
converting from one model to another essentially requires another development effort.

Because there are costs associated with development of any BI model (I’m going to include PowerPivot since it uses the same engine as
Tabular models), selection of the most appropriate model has significant financial implications to the business for which the model is being built. There
have been a number of blog posts and publications discussing the advantages and disadvantages of both Multidimensional and Tabular models. The vast majority of
those publications focus primarily on the technical decision points related to selection of the most appropriate model for project development. Greg Winch
discussed rationales for selection of models in the eGroup blog Tabular vs. Multidimensional Model: what’s right for your business.
Melissa Coates’ (aka SQL Chick) discussed features of PowerPivot, Tabular models and Multidimensional models in her blog Decisions: PowerPivot,
SSAS Tabular, or SSAS Multidimensional Model in SQL Server 2012. Liz Vitt and Scott Cameron (Hitachi Consulting) provide a nice primer that covers the
BI Semantic Model (BISM), Multidimensional modeling and Tabular Modeling (Choosing a Tabular or Multidimensional Modeling
Experience in SQL Server 2012 Analysis Services.) James Serra points out a number of “missing features” in his blog Tabular model: Not
ready for prime time?

The real answer as to which model is the most appropriate is (as the case with most things BI) “it depends.” Before delving
into a lengthy discussion of the factors that should be considered in selection of the model to be used for building a solution, it may be beneficial to engage
in a brief history of the factors that resulted in creation of PowerPivot and the Tabular model. It shouldn’t be surprising that the evolution of these
models is rooted in economics.

There are a number of costs associated with development efforts involving Multidimensional models. These include heavy time
and economic investments in data and dimensional modeling efforts, as well as costs associated with pre-aggregating data, persisting calculations, data storage
and data processing. (For what it’s worth, finding consultants with significant expertise in dimensional modeling is a whole lot easier said than done). Add to
that the fact that mistakes in dimensional models frequently result in direct costs for additional engineering efforts, solutions that don’t meet user needs
or worse yet return incorrect results resulting in business decisions with potentially disastrous financial consequences.

In the meantime, memory costs have continued to decline while hardware performance continually improves. With access to larger
amounts of memory on faster machines, aggregation of data in memory is significantly faster than reading aggregated data from disk. Those factors make
tabular and in-memory models more appealing. Additionally, tabular models are based on more familiar concepts of rows, columns, tables and relationships.
This isn’t to say that mistakes in Tabular modeling aren’t just as costly as mistakes in dimensional modeling. Just imagine the joy of painstakingly
crafting a Tabular model only to find that the most commonly and important query executed against the model consistently takes in excess of 50 minutes to
return a result or that the most crucial DAX calculation returns a value that is incorrect. Business Considerations

I have long lived by the mantra that “the primary purpose of a Business Intelligence solution is
to deliver positive value to the business.” Anything short of that runs a serious risk of becoming a business liability. From the perspective of a
business, that involves providing business people with both the information and the tools necessary for making sound operational and strategic
business decisions.

Feature Set

Selection of a model, therefore, is very much dependent on which features are necessary to deliver information in such a way that it
delivers the desired business value. Multidimensional models, in the current form, have been around for about 11 years and have a very rich feature set. Tabular models, by
way of contrast, were first introduced as PowerPivot 5 years ago. Therefore, it should come as no surprise that Multidimensional models provide a much richer
feature set. The table below displays the feature sets of both Multidimensional and Tabular models.

Feature comparison between Multidimensional, Tabular and PowerPivot Models.

Feature	Multidimensional	Tabular	PowerPivot
Actions	Yes	No	No
Aggregations	Yes	No	No
Calculated Measures	Yes	Yes	Yes
Custom Assemblies	Yes	No	No
Custom Rollups	Yes	No	No
Data Mining	Yes	No	No
Distinct Count	Yes	Yes (via DAX)	Yes (using DAX)
Drill through	Yes	Yes	Yes
Hierarchies	Yes	Yes	Yes
KPIs	Yes	Yes	Yes
Linked objects	Yes	No	Yes (links to Excel tables in the hosting workbook)
Many-to-many relationships	Yes	No (can be modeled using DAX functions)	No (can be modeled using DAX functions)
Parent-child Hierarchies	Yes	Yes (can be modeled using the DAX PATH, PATHLENGTH, LOOKUPVALUE, and PATHITEM functions)	Yes (model using DAX PATH, PATHLEHGTH, LOOKUPVALUE, and PATHITEM functions)
Partitions	Yes	Yes	No
Perspectives	Yes	Yes	Yes
Semi-additive Measures	Yes	Yes	Yes
Translations	Yes	Using DAX	Using DAX
User-defined Hierarchies	Yes	Yes	Yes
Write back	Yes	No	No

Consider the functionalities and features that are not available in Tabular/PowerPivot models. Actions, for instance, are useful when
it is desirable to do something like launch a Reporting Services Report, Provide Custom Drillthrough behavior, or direct to a URL. Custom Assemblies can
be used to implement custom security or functionalities that are not natively present in Analysis Services. Custom Rollups and Unary operators are frequently
used with charts of accounts. Writeback functionality is a very highly desirable feature for performing “What-if” types of analyses.

Available skillset

For a number of reasons, the available skillset is a significant consideration. Part of that is related to
the learning curves that are associated with the two approaches to database design. Amir Netz has compared mastery of the skills associated with
Multidimensional database design and Tabular Database design to the differences between mastering C++ and Visual Basic programming languages.

Multidimensional database design is closely bound to principles of Dimensional Design, involving Dimensions,
Hierarchies, Attributes, Relationships and Measures. Multidimensional database design also makes extensive use of the MDX language. The principles underlying
dimensional modeling differ significantly from those of entity relationship design and resources with dimensional modeling skills tend to be difficult to
secure. Another consideration is that mistakes in dimensional modeling, frequently have significant direct costs due to the need for design changes and
indirect costs resulting from business decisions based on inaccurate or misleading data.

Tabular models, on the other hand, are based on more familiar concepts of Tables, Rows, Columns and
Relationships. Those are concepts are very familiar to DBAs and most Excel users. Add to that, the DAX language is modeled after Excel functions and is
therefore very familiar to Excel users. Multidimensional models, on the other hand, are based on dimensional modeling concepts and Multidimensional
Expressions (MDX). Conversely, selecting the incorrect model for the complexity of the business questions can also be expensive. In either case, it is
important that the architect/consultant/developer have a good grasp of:

Features
Scripting Language to be used
Limitations
Formula Engine
Storage Engine

Complexity of the Business Process being Modeled

The complexity of the model is very closely related to the other considerations for selecting a model. TABULAR
models are relatively easy to implement because they’re based on tables, rows, columns and relationships but have a limited feature set. Multidimensional
models are built around dimensional models and are much more complex to implement but they also provide a richer model because of the feature set. In
short, Tabular models are great for models that will fit into volatile memory and aren’t highly complex while Multidimensional models are more suitable for
highly complex business models and data volumes that won’t fit entirely into memory.

Technical Considerations

Volume of Data

Another significant consideration is data volumes. Tabular models are entirely in-memory and to get decent
performance, the server will need to have at least (2.6 * the size of the data files on disk) to allow for inflation of the files in memory, processing
operations, disaster recovery, etc. Multidimensional models load data into memory on a partition by partition basis and periodically clean unused
partitions from memory, so it’s possible to server larger volumes of data (read that as multiple Terabytes) on commodity hardware.

Feature Set

As indicated previously, Multidimensional models have been around quite a bit longer than Tabular
models. Commonly used features of Multidimensional models that aren’t available in Tabular models include Actions, Aggregations, Custom Assemblies, Custom
Rollups, Writeback functionality. Tabular models also don’t directly Many-to-Many dimensions, or Parent Child dimensions although it’s possible to model either
using DAX expressions. While Multidimensional has the familiar MOLAP, ROLAP and HOLAP storage modes, Tabular models have two separate storage modes (In-Memory
and Direct Query). In-Memory loads the entire data set into memory when the metadata for a database is requested (usually on startup). As of this writing, Direct
Query is only available when using SQL Server as the data source and executes queries against SQL Server.

Needs for Query and Scripting Support

Analysis Services includes MDX, DMX, DAX, XML/A, and ASSL. Support for these languages varies slightly by model type. If query and
scripting language requirements are a consideration, it becomes important to know which scripting languages are supported by the models. Very briefly:

PowerPivot

Uses DAX for calculations and queries

Supports XMLA

Tabular Models

Support DAX Calculations

Support DAX Queries

Support MDX Queries

Supports XMLA

PowerShell

Traditional Analysis Services

Support MDX Calculations

Support MDX Queries

Support XMLA

PowerShell

Security

All Analysis Services solutions can be secured at the database level. More granular security options vary by mode. With PowerPivot,
security is essentially file based. Using either Tabular or Multidimensional models, security is role based although there are differences in the security
implementations. Tabular models use roles to implement row-level security that is based on DAX formulas. Multidimensional models use roles to implement dimension
and cell-level security.

Need for Extensibility

Since Tabular models don’t support Custom Assemblies, it’s simply not possible to add functionalities
using a custom .NET Assembly or COM application. Multidimensional models, conversely, do support custom assemblies which have been used to implement
custom security solutions, as well as add functionalities that aren’t native to MDX. There are a number of examples of such assemblies on the web as well as
CodePlex.

Data Sources

Multidimensional models support a limited set of data sources, which currently include MS Access, SQL Server, SQL
Azure, Parallel Data Warehouse (PDW)/Analytics Platform System (APS), Oracle, DB2, Informix, Teradata and Sybase. With Tabular models, as long as there is an
OLE DB Provider, ODBC Driver, or REST data source, it’s possible to access the data for processing.

Microsoft Analysis Services using ROLAP storage in Teradata.

John Desch MSFT — Tue, 23 Dec 2014 20:45:00 GMT

I recently had the opportunity to work with a customer who was interested in implementing a Microsoft SQL Server 2012 Analysis Services database, using ROLAP storage, with Teradata as the relational data source. With a week to build a Proof of Concept (POC) solution using Microsoft SQL Server Analysis Services database and having had previous encounters with Analysis Services using ROLAP storage with Teradata as the source relational database, I was headed into familiar territory. Knowing that MOLAP storage typically provides faster query response, my first thought and initial objective was to try to steer the customer away from ROLAP storage and toward MOLAP storage.

I’ll state here that I have a definite bias towards MOLAP storage and a somewhat stronger bias toward MOLAP when any relational data source is being used in conjunction with a Managed Code OLE DB provider. The reasons for that bias are related to query performance and rest with two primary considerations. The first is that regardless of the source of the relational data, MOLAP storage typically performs significantly better than ROLAP storage. The second basis for my bias is that in 64-bit environments, when connecting to Teradata, the available Teradata OLE DB provider is a .NET Managed code provider. Because this is a managed code OLE DB provider being used in a native code application (Analysis Services), there is a significant amount of additional marshalling that must occur to handle things like .NET garbage collection. The additional marshalling that must occur results in somewhat slower performance than is the case with native code OLE DB providers when retrieving data from the relational data source.

After a reasonably short conversation with the customer, it was readily apparent that the focus of interest was minimizing processing times and having data available in real-time or near real-time. Given that MOLAP storage would have required very frequent processing with inevitable latencies, it quickly became apparent that ROLAP storage would necessarily be part of the solution. So much for the original action plan and now I had serious questions about whether or not this POC could be completed in the time span of one week. Now it was time to either concede defeat or come up with a new plan of attack.

Having been down this path previously, I had a few resources from which to draw:

I’ve dealt with similar scenarios multiple times in the past, so I know what potential pitfalls exist.
I’ve been involved with support for Microsoft SQL Server Analysis Services since 2000.
I’m more familiar than I probably should be with the whitepaper Improve your OLAP Environment With Microsoft and Teradata, written by Rupal Shah and Richard Tkachuk in 2007.

While the whitepaper by Shah and Tkachuk was written in relation to a specific implementation, it does list a number of well-defined actions which provide significant performance benefits in a ROLAP environment. Those include the following:

Use normal ETL Processing to load or update base tables in an Enterprise Data Warehouse
Build a Semantic Layer to contain analytic specific content with the following characteristics:
1. A Star or Snowflake schema.
2. Primary and Foreign Keys defined as not nullable and not compressible.
3. Dimension table Primary Keys use the UNIQUE constraint.
4. Use ID rather than Name or Description columns as Primary Keys.
5. Ensure that measures in any Fact Tables are a numeric data type.
6. Implement Referential Integrity on Primary Key/Foreign Key columns.
7. Consider creation of secondary indexes on FACT columns.
8. Take advantage of partitioning with Partitioned Primary Indexes to improve performance.
Create Teradata Aggregate Join Indexes (AJIs) to facilitate delivery of data in a timely manner.
Check the relational queries generated by Analysis Services against the defined AJIs and check the request using the Teradata Explain command to ensure that the AJI is included in the query plan.

Since using ROLAP storage will result in relational queries being sent from Analysis Services to Teradata, all of that makes a great deal of sense. At this point, one could confidently open SQL Server Data Tools (SSDT) and begin designing an Analysis Services database using Teradata as the data source and live happily ever after. Unfortunately, there are still a few things that Shah and Tkachuk didn’t really touch on in their whitepaper. One of the first things that we want to do is configure the server such that it’s in a state that is consistent with information from the SQL Server 2008 R2 Analysis Services Operations Guide. Granted, this is a bit dated and some of the configuration settings are different in the SQL Server 2012 and 2014 releases (particularly the MemoryHeapType and HeapTypeForObjects settings).

Out of the box, Microsoft Analysis Services is configured and optimized for SINGLE USER QUERY EXECUTION. Essentially, that means that it’s quite possible for a single long running query that is bottlenecked in Storage Engine (in this case retrieving data from Teradata) can potentially block other
queries. To change that and configure the server for MULTI-USER QUERY EXECUTION:
1. Locate the msmdsrv.ini file (default location is C:\Program Files\Microsoft SQL Server\MSASXX.MSSQLSERVER\OLAP\Config for a default install and C:\Program Files\Microsoft SQL Server\MSASXX.\OLAP\Config for a named instance).
2. Create a backup of the file, just in case you generate malformed XML while editing the file.
3. Open the file with NotePad or some other text editor and find the following XML tags:
4. Change the values FROM:
  
  -1
  3
  
  TO:
  
  1
  0
Locate the and XML tags in the msmdsrv.ini file and change FROM:

8192
8192

TO:

65536
65536
If you happen to be working with a Microsoft SQL Server 2008 or 2008 R2 instance of Analysis Services, I’d strongly recommend upgrading to 2012 or 2014. If upgrading a 2008 or 2008 R2 instance isn’t an option, then you’ll want to:
1. Locate the following XML tags in the msmdsrv.ini file
2. Change the values FROM:
  
  1
  1
  
  TO:
  
  2
  0
Save then close the edited msmdsrv.ini file.
Stop and restart the Analysis Server.

To get maximum query performance, it’s going to be necessary to build appropriate AJIs in Teradata and ensure that the statistics are current. Now it’s time to consider some of the things that your mother never taught you about working with Analysis Services using ROLAP Storage with Teradata as the relational source. A close working relationship with the Teradata DBA(s) becomes essential because query patterns tend to change over time. That means that the AJIs in the Teradata semantic layer must be periodically tuned. If the environment is such that users are accessing data in non-parameterized reports and not executing ad-hoc queries, then it’s quite easy to capture the Progress Report End events in a Profiler trace or via Teradata DBQL access logs. Things become a bit dicey when users are viewing data in interactive reports or are executing interactive queries that apply filters and this is where things get interesting.

When users are interacting with the data in parameterized reports or executing ad-hoc queries that apply filters, Analysis Services submits a set of one or more parameterized SQL statements to Teradata for execution. When the queries are sent to Teradata via the OLE DB provider, the Parameters are represented by a “?” as a parameter placeholder. This makes tuning the AJIs somewhat challenging in that neither the OLE DB provider nor the Teradata DBQL Logs expose the actual parameter values. Without knowing which partitions are being accessed, tuning the AJIs becomes pure guess work.

You can identify long running SQL statements relatively easily by:

Start a Profiler trace to capture the Query Begin, Query End, and Progress Report End events.
Run the set of MDX queries that generated poorly performing parameterized SQL queries.
Save the Profiler trace to a table in SQL Server
Execute the following SQL Query:

SELECT ROWNUMBER, EVENTCLASS, EVENTSUBCLASS, TEXTDATA, DURATION FROM
WHERE EVENTCLASS=6 AND EVENTSUBCLASS=25 ORDER BY DURATION
DESC

While that will give you some information regarding long running SQL statements, it will not provide information related to actual parameter values. In order to capture the actual parameter values that are being passed from Analysis Services to Teradata, we have to make a change to the cartridge that is used by Analysis Services when connecting to a Teradata server. NOTE: Everything you are going to read from this point on is totally unsupported by Microsoft.

Locate the trdtv2r31.xsl file (default location for a default installation is C:\Program Files\Microsoft SQL Server\MSAS11.MSSQLSERVER\OLAP\bin\Cartridges)
Make a copy the trdtv2r31.xsl file from the cartridges subdirectory and save it to another directory in unedited form.
Manually edit the trdtv2r41.xsl cartridge file in the cartridges subdirectory and change
FROM:

yes

TO:

no.
Stop and re-start the Analysis Services service. (By The Way, Congratulations. Your server is now in a totally unsupported state, so if you encounter a problem you’ll need to restore the unedited copy of the trdtv2r31.xsl file before contacting Microsoft Support for assistance).
Start a Profiler trace to capture the Query Begin, Query End, and Progress Report End events.
Run the set of MDX queries that generated poorly performing parameterized SQL queries.
Extract the SQL Statements in the Progress Report End EventClass and ExecuteSQL Event Subclass (EventClass =6 and EventSubclass=25)
Send those SQL Statements to the Teradata DBA(s) so they can tune the AJIs.
Stop the Analysis Services service
Replace the modified trdtv2r31.xsl file with the original and unedited version of the file.
Restart the Analysis Services service
NOTE: You server is now back in a supported state
Test the queries after the AJIs have been tuned.

After working through all of these challenges, we were ready to do some testing. Initial testing using a third party application directly against Teradata with no AJIs defined yielded average query response times of 00:02:05 to retrieve and render data. After building an OLAP database on top of Teradata and using Excel to
execute queries, again with no AJIs defined, average query response times were 00:02:03. The difference was not considered to be significant. After using the method describe above to identify parameter values and build a set of AJIs, with a cold Analysis Services data cache, query response times averaged 00:00:35
and with a warm Analysis Services data cache, query response times averaged 00:00:00.015.

Needless to say, we were able to successfully build the POC database in the span of a week and query performance actually exceeded customer expectations.

Using DMVs and XMLA commands to force a disconnect from an Analysis Server

John Desch MSFT — Fri, 14 Nov 2014 16:06:00 GMT

At one time or other, just about every DBA charged with administering a Microsoft SQL Server Analysis Services server has encountered a situation that necessitated forcing a disconnect from either a Multidimensional or Tabular instance of Analysis Services. In the relational world of SQL Server, that's relatively easily accomplished by using SP_WHO, to determine the SPID to be terminated followed by execution of a KILL command. Accomplishing the same thing with Analysis Services is a bit more complicated, but can be easily accomplished given a little familiarity with the Dynamic Management Views (DMVs) and XMLA commands. .

When a user connects to an Analysis Server, two objects are created on the server and both are easily detected using DMV queries. One of those objects is a Connection and the connections can be discovered using the following DMV query:

SELECT CONNECTION_ID, CONNECTION_USER_NAME, CONNECTION_IMPERSONATED_USER_NAME,

CONNECTION_HOST_APPLICATION, CONNECTION_LAST_COMMAND_ELAPSED_TIME_MS, CONNECTION_IDLE_TIME_MS

FROM $SYSTEM.DISCOVER_CONNECTIONS

That give us some information, but not quite enough yet to terminate a connection, because we need to get the SPID that is associated with the connection. The SPID is actually contained within the Session that is associated with the connection, and we can get that with the following DMV query.

SELECT SESSION_SPID, SESSION_CONNECTION_ID, SESSION_USER_NAME, SESSION_LAST_COMMAND, SESSION_LAST_COMMAND_ELAPSED_TIME_MS

FROM $SYSTEM.DISCOVER_SESSIONS

The SESSION_SPID column gives us the SPID associated with each connection and the SESSION_CONNECTION_ID maps directly to the CONNECTION_ID that was returned from the query against the $SYSTEM.DISCOVER_CONNECTIONS DMV. We definitely don't want to terminate SPID 1652, which is being used to run the DMV queries, but we do have a list of SPIDS from which we can select. We can tell from the output that SPID 3928 is currently idle and we want to terminate the connection because we know that the user has gone for the day and will not be returning for a couple of days. To do that, and this happens to be executing against a Tabular server, we'll execute the following XMLA command:

<Cancelxmlns=https://schemas.microsoft.com/analysisservices/2003/engine>

<CancelAssociated>1CancelAssociated>

Cancel>

If we re-execute the DISCOVER_CONNECTIONS and DISCOVER_SESSIONS DMV queries, we'll find that connection 192 has been closed.

Using SQL Server to Analyze Analysis Services Profiler Traces from Tabular Models.

John Desch MSFT — Thu, 26 Jun 2014 23:09:00 GMT

My last blog post was focused on using SQL Server to analyze profiler traces against Multidimensional Analysis Services databases, so I though it only appropriate to follow-up with a post on using SQL Server to analyze profiler traces from a Tabular Analysis Services database. In July of 2013, John Sirmon, Greg Galloway (Artis Consulting), Cindy Gross and Karan Gulati published the Performance Tuning of Tabular Models in SQL Server 2012 Analysis Services whitepaper (https://msdn.microsoft.com/en-us/library/dn393915.aspx). They described the process that they used for determining whether a bottleneck was in Storage Engine for Formula Engine. I tried that approach (basically did a cut and paste of the SQL Query that begins on page 35) with a brand new profiler trace and was more than a bit surprised to see the user name associated with the fastest query in the trace associated with the slowest query in the trace. I was also a bit surprised to find that the Storage Engine time values exceeded the durations for the queries and that the duration reported was the sum of the durations of all eight (8) queries that were captured in the trace..

Never one to be discouraged, it didn't take me too long to figure out that this is basically the same task as analyzing profiler traces from Multidimensional databases. The big difference is the events that indicate Multidimensional Storage Engine events and those that indicate Vertipaq Engine events. In the case of multidimensional queries, Storage Engine events and durations are associated with EventClass 11 (Query Subcube) and EventSubclass 2 (Non-cache data). With Tabular models, VertiPaq engine events and durations are associated with EventClass 83 (VertiPaq SE Query End), with particular interest in EventSubclass 0 (VertiPaq Scan). Taking that bit of information, I came up with the following SQL Query that gave me exactly the information that I was looking for:

with ProcAnalysis (Rownumber, EventClass, EventSubClass, TextData, QueryDuration, SETime, MinRow, Bottleneck, NTUserName, ConnectionID, ZScore)
AS
(
select a.Rownumber, pec.name as EventClass, pes.name as EventSubclass, cast(a.TextData as nvarchar(max)) as TextData, a.duration,
(select sum(duration) from ProfilerTraceTable b where b.eventclass=83 and b.eventsubclass=0 and b.connectionID=a.ConnectionID and b.rownumber between
(select max(rownumber) from ProfilerTraceTable c where c.eventclass=9 and c.rownumber
(select max(rownumber) from ProfilerTraceTable c where c.eventclass=9 and c.rownumber
case when (select sum(duration) from ProfilerTraceTable b where b.eventclass=83 and b.eventsubclass=0 and b.connectionID=a.ConnectionID and b.rownumber between
(select max(rownumber) from ProfilerTraceTable c where c.eventclass=9 and c.rownumbera.duration*.5 then 'VertiPaq Engine'
else
'Formula Engine'
end as Bottleneck,
a.NTUserName, a.ConnectionID,
case when (select stdev(b.duration) from ProfilerTraceTable b where a.eventclass = b.eventclass and a.eventsubclass=b.eventsubclass and duration is not null) = 0
or (select avg(b.duration) from ProfilerTraceTable b where a.eventclass = b.eventclass and a.eventsubclass=b.eventsubclass and duration is not null)=0 then 0
else
(a.duration-(select avg(b.duration) from ProfilerTraceTable b where a.eventclass = b.eventclass and a.eventsubclass=b.eventsubclass and duration is not null))
/
(select stdev(b.duration) from ProfilerTraceTable b where a.eventclass = b.eventclass and a.eventsubclass=b.eventsubclass and duration is not null)
end
as ZScore
from ProfilerTraceTable a
inner join ProfilerEventClass pec on a.EventClass=pec.EventClassID
inner join ProfilerEventSubclass pes on a.EventClass=pes.EventClassID
and a.EventSubclass=pes.EventSubClassID
)
select RowNumber, EventClass, EventSubclass, TextData, QueryDuration, SETime, Bottleneck, NTUserName, Connectionid, ZScore
From ProcAnalysis
where EventClass = 'Query End'
-- and cast(zscore as float)>0.01
order by queryduration desc, ZScore desc

Processing Analysis Services. Notes on the Care and Feeding of an OLAP Database

John Desch MSFT — Thu, 26 Jun 2014 22:05:00 GMT

Anyone who has had the opportunity to work with Microsoft Analysis Services, in multidimensional mode, knows that processing is a series of steps that are used to load data from a relational data source into a model. By default, processing occurs when a solution is deployed or re-deployed to the server from BI Development Studio or SQL Server Data Tools. Processing can also be accomplished using SQL Server Management Studio by using SQL Server Integration Services and SQL Server Agent. What actually happens during processing is dependent on several factors that need to be carefully considered. Structural changes to models, dimension design characteristics, processing options selected, configuration settings, and other executing operations are all important factors that can impact processing performance and the outcome of processing operations.

Structural Changes to a model

If there have been structural changes to a model, it is going to be necessary to process any objects that are impacted by the structural change. Just to be clear, a structural change is going to result in changes to the metadata as well as the structure of the data and aggregations.

Adding a dimension to a database requires processing the newly added dimension.
Adding a dimension to a cube necessitates processing the dimension and any cubes to which it has been added.
Removing a dimension from a cube or database requires reprocessing the cubes in which the dimension was used
Changing the compatibility level of a dimension requires that the dimension and any cubes in which it is used be reprocessed

Dimension Design Characteristics

The way in which attribute relationships are defined within a dimension has a very distinct impact on processing aggregations. By default, all attributes are directly related to the granularity attribute using flexible attribute relationships. That essentially means that members are expected to change positions and potentially levels within the structure of a dimension, which results in creation of flexible attribute relationships. Flexible attribute relationships are fine in situations where members of a dimension are likely to change positions, but they can also result in some performance issues. Executing a ProcessUpdate on any dimensions that are designed with flexible attribute relationships will result in flexible aggregations associated with that dimension being dropped. That results in some performance degradation because without the flexible aggregations, the server is forced to scan partitions rather than use the functional equivalent of indexes. If it is necessary to use flexible attribute relationships, consider the following options:

When executing ProcessUpdate, check the “Process affected objects” checkbox, which will process the objects in a single transaction and rebuild the flexible aggregations. This may take somewhat longer for processing to complete.
After executing ProcessUpdate, execute a ProcessIndex on the measure group(s) that use the affected dimension.
Configure the ProcessingMode property of measure groups that use the dimensions with flexible attribute relationships for LazyAggregation.

Processing indexes - Adjust AggregationMemoryLimitMin and AggregationMemoryLimitMax

Processing Options

There are a number of processing options available and not all options apply to all objects. Another consideration is that different processing operations apply to different objects stored in an Analysis Services database, as outlined in the table below.

Processing Type	Cubes	Databases	Dimensions	Measure Groups	Partitions	Mining Models	Mining Structures
Add			X	X	X
Clear	X	X	X	X	X	X	X
Data	X		X	X	X
Default	X	X	X	X	X	X	X
Full	X	X	X	X	X	X	X
Index	X		X	X	X
Structure	X						X
Update			X

Process Add

For dimensions, adds new members and updates dimension attribute captions and descriptions.
For measure groups and partitions, adds newly available fact data and process only to the relevant partitions.

Process Clear

Drops the data in the object specified and any lower-level objects. Executing a ProcessClear at the database level would drop the data for all dimensions, cubes, measure groups, partitions, and mining models contained within the database. Executing a ProcessClear on a cube would remove all data from the measure groups and partitions contained in the cube.

Process Data

Drops the data in partitions and re-populates with data from the data source. Using this processing option does not build any aggregations or indexes..

Process Default

This processing option detects the state of the database object or objects that are to be processed and performs any actions necessary to render the object(s) to a fully processed state. In the event that a structural change has been made to a database or the database, cube, etc. is unprocessed, ProcessDefault will execute a ProcessFull on the object. For example, if a partition were changed from a table binding to query binding, ProcessDefault would execute a ProcessFull on the partition. Likewise, if a ProcessDefault were executed on an unchanged partition, the action would process data and indexes on the partition. .

Process Full

Processes an Analysis Services object and all the objects that it contains. When Process Full is executed against an object that has already been processed, Analysis Services drops all data in the object, and then processes the object. This kind of processing is required when a structural change has been made to an object, for example, when an attribute hierarchy is added, deleted, or renamed.

Process Index

This processing creates or rebuilds indexes and aggregations for all partitions contained by the object being processed. When executing a ProcessIndex on a cube, aggregations an indexes are built for all of the partitions contained within that cube. Likewise, when a ProcessIndex is executed on a Measure Group, aggregations and indexes are built for all of the partitions contained within that measure group. I probably should point out that this option is not valid for objects that are in an unprocessed state, so attempting to execute a ProcessIndex on a cube that is in an unprocessed state will generate an error. Creates or rebuilds indexes and aggregations for all processed partitions. If a cube contains one or more dimensions built with flexible attribute relationships and the cube is not configured for Lazy Processing, executing a ProcessIndex is an action that should be seriously considered after a ProcessUpdate on a dimension that contains flexible attribute relationships.

Process Update

This processing option forces the server to re-read the source data for the dimension and update of dimension attributes. It's important to bear in mind that executing a ProcessUpdate on dimensions that have been built with flexible attribute relationships will cause all of the flexible aggregations and indexes on related partitions to be dropped. This can result in potential query performance issues and it's often desirable to either check the “Process affected objects” checkbox or seriously consider executing a ProcessIndex on the affected cube(s) after executing the ProcessUpdate.

Process Structure

ProcessStructure really only applies to Cubes and Mining Structures. Applied to a Cube, If the cube is in an unprocessed state, the server will process the metadata for the cube and all of the dimensions that are used within the structure of the cube. Applied to Mining model structures, ProcessStructure populates the mining structure with source data but does not actually load data into the mining models.

ErrorConfiguration Settings

These configuration settings determine the actions taken when errors are encountered during processing. Using the default error configuration, processing will stop on an error. One can, however, select the "Use custom error configuration" option, which allows selection of values for the following actions to control error-handling behavior:

Calculation Error. This setting determines the action to be taken when an error is encountered in the calculation script. The options are:

Ignore Error, which is the default action
ReportAndContinue
ReportAndStop

KeyDuplicate. This configuration option determines what action is taken when a duplicate key is encountered in a dimension.The options are:

Ignore Error.This is an option, which I generally consider to be very undesirable because it tends to mask referential integrity and data quality issues in the underlying relational data source.
ReportAndContinue
ReportAndStop. This is the default option and is very commonly implicated in processing failures, although the underlying problem is a data quality issue in the relational source database.

KeyErrorAction. If a key value does not yet exist in a record, one of these actions is selected to occur:

Convert to unknown. The key is interpreted as an unknown member. This is the default setting.
Discard record.

KeyErrorLimit. This is an integer value that sets the maximum number of key errors that can occur before the Key Error action is invoked.

KeyErrorLimitAction. This setting determines the action that is taken when the number of key errors reaches the value of the KeyErrorLimit configuration setting. The options are:

StopProcessing
StopLogging

KeyErrorLogFile. This is very simply a path to a text file to which errors will be logged.

KeyNotFound. This setting determines the action that Is taken when a key is not found. The options are:

Ignore Error. This is probably acceptable for a development environment, but is probably not a good selection for a production environment due to the potential for masking data quality issues.
ReportAndContinue
ReportAndStop

NullKeyConvertedToUnknown. This setting determines the action taken when a null key is converted to "Unknown". The default is to Ignore the error. The options are:

Ignore Error
ReportAndContinue
ReportAndStop

NullKeyNotAllowed. This setting determines the action taken when a null key is encountered during processing. The default is to Ignore the error. The options are:

Ignore Error
ReportAndContinue
ReportAndStop

Other Executing Operations

For one reason or other, operations that are being executed by a server are frequently overlooked with respect to processing. Unfortunately, this can sometimes result in deadlocks and processing failures. Deadlocks can occur in a number of different scenarios. One of the most common scenarios that results in processing failure is related to attempts to process objects in two separate but concurrent transactions. The problem in this scenario is that Analysis Services uses a two phase commit process during processing. As part of the commit, the server must update the Master.VMP file, which contains the master version map for all objects on the server. Updating that file requires a server level lock and when two objects are being processed in parallel and both are ready to commit the transaction, both must update the Master.VMP file. This results in one of the processing transactions becoming a deadlock victim and the transaction being rolled back. In this scenario, the best option to consider is to use the DMV query SELECT * FROM $SYSTEM.DISCOVER_SESSIONS and examine the SESSION_LAST_COMMAND column to check for the presence of a processing command before initiating a processing operation on another object and execute the processing operations in sequence rather than in parallel.

Using SQL Server to Analyze Analysis Services Profiler Traces

John Desch MSFT — Mon, 02 Jun 2014 05:36:00 GMT

In 2007, Carl Rabeler and Eric Jacobsen wrote a fairly detailed white paper entitled Identifying and Resolving MDX Query Performance Bottlenecks in SQL Server 2005 Analysis Services (https://www.microsoft.com/en-us/download/details.aspx?id=661). In that white paper, they describe a method of determining if the bottleneck is in Storage Engine or Formula Engine that involves adding the elapsed time for each Query Subcube event in a SQL Server Profiler trace. What the white paper doesn't explicitly state is that determining the amount of time spent in Storage Engine involves adding up the elapsed time for each Query Subcube event that is associated with a particular ConnectionID and a particular Query. One approach to analyzing SQL Server Profiler Traces captured from an Analysis Services Server has been to first save the trace to a SQL Server table and then execute a series of SQL queries similar to the following:

-- First get a list of the Query End events ordered by duration in descending order

SELECT ROWNUMBER, EVENTCLASS, EVENTSUBCLASS, TEXTDATA, CONNECTIONID, DURATION FROM WHERE EVENTCLASS=10 ORDER BY DURATION DESC

-- Note the ConnectionID and Rownumber of the Long running Query so that we can get the RowNumber for the corresponding Query Begin events

SELECT TOP 2 ROWNUMBER, EVENTCLASS, EVENTSUBCLASS, TEXTDATA, CONNECTIONID, DURATION FROM

WHERE EVENTCLASS IN (9,10) AND CONNECTIONID= AND ROWNUMBER<=

ORDER BY ROWNUMBER DESC

-- Note the RowNumber of the Query Begin event and then sum the durations for the Query Subcube events for the specific query that was executed on a specific connection

SELECT SUM(DURATION) AS SE_TIME FROM WHERE EVENTCLASS=11 AND CONNECTIONID=

AND ROWNUMBER BETWEEN AND

The resulting set of queries starts to look something like the following:

select rownumber, eventclass, eventsubclass, textdata, connectionid, duration, starttime, currenttime from MYTABLE where EventClass=10 order by duration desc

select top 2 rownumber, eventclass, eventsubclass, textdata, connectionid, duration, starttime, currenttime from MYTABLE where
EventClass in (9,10) and ConnectionID=398 and RowNumber <=12280 order by RowNumber desc

select SUM(duration) as SE_TIME from MYTABLE where EventClass=11 and ConnectionID=398 and RowNumber between 11800 and 12280

You can then use the rule of thumb that any query where the value returned for SE_TIME exceeds 30% of the query duration is bottlenecked in Storage Engine. It's a great methodology for quickly identifying where performance issues reside in an MDX query. This methodology however, assumes that the person doing the analysis knows that EventClass 9 is the Query Begin event, EventClass 10 is the Query End event, and EventClass 11 is the Query Subcube event.

The approach described above, however, has a couple of pitfalls.

1. If there are large numbers of "long running" MDX queries captured in the Profiler trace, execution of a large number of SQL queries becomes necessary

2. This approach is only useful for examining bottlenecks in MDX queries and can't really be used to analyze processing performance bottlenecks, because a substantial amount of work is done in parallel.

With those limitations in mind, I decided to fall back on a background in statistics and try a somewhat different approach that involves a bit of SQL Algebra. The first thing I wanted to do was build a clean SQL Server database that would contain the data from Profiler traces on which I wanted to perform analyses. Pretty simple task, just launch good old SSMS and execute the following SQL commands:

CREATE DATABASE OLAPANALYSIS

USE OLAPANALYSIS

So now I've got a database to store my data in but I want to do more than just analyze performance issues involving MDX queries. I also want to be able to analyze processing performance issues, which means that I need to know all of the different EventClass and EventSubclass codes represent. Because I really have no greate burning urgent desire to memorize close to 400 numeric codes, I'm just going to use a bit of XQuery magic and import the data from the Trace Definition XML file into a pair of tables using the following SQL statements:

CREATE TABLE ProfilerEventClass (EventClassID int not null, Name nvarchar(50),Description nvarchar(500))
GO

-- Since I'm using my Denali instance of SQL Server 2012 Analysis Services, I'm going to use the trace
-- definition file that's located in the bin\Resoureces\1033 sub-directory for US English
declare @xml as XML
SELECT @xml = x.y
FROM OPENROWSET( BULK 'C:\Program Files\Microsoft SQL Server\MSAS11.DENALI\OLAP\bin\Resources\1033\tracedefinition110.xml', SINGLE_CLOB ) x(y)
insert into ProfilerEventClass(EventclassID, Name, Description)
select c.c.value('(ID/text())[1]','INT')EventClassID,
c.c.value('(NAME/text())[1]','VARCHAR(50)')Name,
c.c.value('(DESCRIPTION/text())[1]','VARCHAR(500)')Description
from @xml.nodes('TRACEDEFINITION/EVENTCATEGORYLIST/EVENTCATEGORY/EVENTLIST/EVENT') as c(c)

CREATE TABLE ProfilerEventSubclass (EventClassID int not null, EventSubclassID int not null, Name nvarchar(50))
GO

declare @xml as XML
SELECT @xml = x.y
FROM OPENROWSET( BULK 'C:\Program Files\Microsoft SQL Server\MSAS11.DENALI\OLAP\bin\Resources\1033\tracedefinition110.xml', SINGLE_CLOB ) x(y)
insert into ProfilerEventSubclass(EventClassID, EventSubclassID, Name)
select c.c.value('(../../../../ID/text())[1]','INT')EventClassID,
c.c.value('(ID/text())[1]','INT')EventSubClassID,
c.c.value('(NAME/text())[1]','VARCHAR(50)')Name
from @xml.nodes('TRACEDEFINITION/EVENTCATEGORYLIST/EVENTCATEGORY/EVENTLIST/EVENT/EVENTCOLUMNLIST/EVENTCOLUMN/EVENTCOLUMNSUBCLASSLIST/EVENTCOLUMNSUBCLASS') as c(c)
GO

So now I've got a complete set of the EventClass and EventSubclass codes as well as the Names associated with each. Now I can save my Profiler trace to a table and start to perform some analyses, but since the profiler trace I'm interested in contains close to 1000 long running MDX queries in addition to some processing operations I really don't care to spend several hours executing multiple SQL queries to determine the bottleneck for each of those long running queries. That means it's time for a little bit of CTE and some correlated SQL queries with a bit of a twist. The twist, in this case is that I'm going to calculate a Z Score so that I can look at the duration data on the basis of how much it deviates from the average duration for a given event.

with ProcAnalysis (Rownumber, EventClass, EventSubClass, TextData, QueryDuration, SETime, MinRow, Bottleneck, ConnectionID, ZScore)
AS
(
select a.Rownumber, pec.name as EventClass, pes.name as EventSubclass, cast(a.TextData as nvarchar(max)) as TextData, a.duration,
(select sum(duration) from changinddims b where b.eventclass=11 and b.connectionID=a.ConnectionID and b.rownumber between
(select max(rownumber) from changinddims c where c.eventclass=9 and c.rownumber (select max(rownumber) from changinddims c where c.eventclass=9 and c.rownumbercase when (select sum(duration) from changinddims b where b.eventclass=11 and b.connectionID=a.ConnectionID and b.rownumber between
(select max(rownumber) from changinddims c where c.eventclass=9 and c.rownumbera.duration*.3 then 'Storage Engine'
else
'Formula Engine'
end as Bottleneck,
a.ConnectionID,
case when (select stdev(b.duration) from changinddims b where a.eventclass = b.eventclass and a.eventsubclass=b.eventsubclass and duration is not null) = 0
or (select avg(b.duration) from changinddims b where a.eventclass = b.eventclass and a.eventsubclass=b.eventsubclass and duration is not null)=0 then 0
else
(a.duration-(select avg(b.duration) from changinddims b where a.eventclass = b.eventclass and a.eventsubclass=b.eventsubclass and duration is not null))/(select stdev(b.duration) from changinddims b where a.eventclass = b.eventclass and a.eventsubclass=b.eventsubclass and duration is not null)
end
as ZScore
from changinddims a
inner join OlapAnalysis.dbo.ProfilerEventClass pec on a.EventClass=pec.EventClassID
inner join OlapAnalysis.dbo.ProfilerEventSubclass pes on a.EventClass=pes.EventClassID
and a.EventSubclass=pes.EventSubClassID
)
select RowNumber, EventClass, EventSubclass, Textdata, QueryDuration, SETime, MinRow, Bottleneck, Connectionid, ZScore
From ProcAnalysis
where EventClass = 'Query End'
-- To look at queries the 50% of queries that are above average, use a Z Score value of 0
-- To look at the top 32% of long running queries, use a Z Score value of 1
and cast(zscore as float)>0
order by queryduration desc, ZScore desc

But now I want to take a look at some issues related to processing operations using a variant of the same methodology. Again using a CTE query with correlated sub-queries, I'm simply going to execute the following SQL statement:

with ProcAnalysis (EventClass, EventSubClass, ConnectionID, ObjectPath, TextData, Duration, AverageVal, StandardDev, ZScore)
AS
(
select pec.name as EventClass, pes.name as EventSubclass, a.ConnectionID, a.ObjectPath, cast(a.TextData as nvarchar(max)) as TextData, a.duration,
(select avg(b.duration) from changinddims b where a.eventclass = b.eventclass and a.eventsubclass=b.eventsubclass and duration is not null) as avgval,
(select stdev(b.duration) from changinddims b where a.eventclass = b.eventclass and a.eventsubclass=b.eventsubclass and duration is not null) as stddeval,
case when (select stdev(b.duration) from changinddims b where a.eventclass = b.eventclass and a.eventsubclass=b.eventsubclass and duration is not null) = 0
          or (select avg(b.duration) from changinddims b where a.eventclass = b.eventclass and a.eventsubclass=b.eventsubclass and duration is not null)=0 then 0
else
-- Calculate a Z Score to determine deviation from the mean
(a.duration-(select avg(b.duration) from changinddims b
where a.eventclass = b.eventclass and a.eventsubclass=b.eventsubclass and duration is not null))
   /(select stdev(b.duration) from changinddims b where a.eventclass = b.eventclass and a.eventsubclass=b.eventsubclass and duration is not null)
end
as ZScore
from changinddims a
inner join OlapAnalysis.dbo.ProfilerEventClass pec on a.EventClass=pec.EventClassID
inner join OlapAnalysis.dbo.ProfilerEventSubclass pes on a.EventClass=pes.EventClassID
   and a.EventSubclass=pes.EventSubClassID
)
select EventClass, EventSubclass, Connectionid, ObjectPath, TextData, Duration, AverageVal, StandardDev, ZScore
From ProcAnalysis
where EventClass != 'Notification' and duration is not null and duration < (select max(duration) from ProcAnalysis)
-- To look at the top 50% of long running operations, use a Z Score value of 0
-- To look at the top 32% of long running operations, use a Z Score value of 1
and cast(zscore as float)>1
order by duration desc, ZScore desc

That query returns the following data set:

Using SSIS to Backup and Restore Extremely Large OLAP Databases

John Desch MSFT — Fri, 18 Apr 2014 15:05:00 GMT

Working in the field of Business Intelligence I get the opportunity to work with some really large (read that as multi-terabyte) OLAP databases. Multi-terabyte OLAP databases, while not yet common place, are being seen with greater frequency and they do present a few interesting challenges to developers and administrators. Performance tuning is one of the more obvious challenges, leading to discussions related to selection of the most appropriate storage mode and how to best partition the data. Purely from the perspective of query performance, MOLAP storage is going to provide better performance than would be expected with HOLAP or ROLAP storage. At these sizes, I/O throughput is a concern and there are definite benefits to using the StorageLocation property to distribute partition data across multiple disks.

A less obvious aspect of performance tuning involves the ability to backup and restore these large databases before Mr. Murphy applies his law and it becomes necessary to recover from some form of disaster. Databases can become inaccessible for a number of reasons, including but not limited to hardware failures and BI developers with administrator permissions fully processing dimensions. The prospect of failure and the need to recover from a disaster gives administrators of OLAP databases some really good reasons to be involved in planning and testing of backup and recovery operations.

For reasons I’ll explore in a bit, Analysis Services backup and restore operations on multi-terabyte databases do not occur with lightning speed. Likewise, if it becomes necessary to execute a full process of a multi-terabyte database there’s a pretty good chance that it’s going to take more than just a few hours. Just imagine the fun of explaining to the company CEO, CFO, and CIO that you’ll have their production database back online in a couple of weeks. Therefore, the ability to restore a really big database to a functional state within a reasonable period of time is probably more important than the ability to create a backup in a reasonably short period of time. At least the discussion with the CEO, CFO, and CIO will be substantially less painful if you can say something to the effect of “We should have the database fully restored in a few hours.”

I recently had the opportunity to work with a customer who had a 35 hour window in which to create a backup of an OLAP database that occupied nearly three terabytes on disk. Because of the size of the database, the partitioning strategy involved distributing the data across eight separate LUNs. The reason for the 35 hour window was that a processing job was scheduled to begin execution on Sunday evenings at 11:00 PM. Because the database was in use, the backup job was scheduled to begin on Saturdays at noon. The problem was that the backup job would be terminated after 35 hours when the scheduled processing operation began executing. Since this was a production system, it was absolutely essential that the customer have a database backup and a plan to restore the data in the event of a disaster.

In the absence of a database backup, the alternative was to have the database remain offline for a period of nine (9) days, possibly longer, to allow the database to be fully processed.So what are the options for implementing backup and recovery with very large OLAP databases as well as the drawbacks of each approach? Let’s take a look the following options:

SAN Snapshot
Backup and Restore
Re-deploy and fully process
Synchronization

1. SAN Snapshot: This is an option that has been explored in several scale-out scenarios solely for the purpose of moving metadata and data files from a dedicated processing server to one or more query servers. (see Carl Rabeler’s whitepaper entitled “Scale-Out Querying with Analysis Services Using SAN Snapshots” https://www.microsoft.com/en-us/download/details.aspx?id=18676). SAN Snapshots are fantastic for backing up one or more LUNs for storage locally or at another site. There are essentially two types of SAN Snapshot. The While this is greatly simplified, it’s sufficient to know that a Copy-on-Write snapshot creates a snapshot of changes to stored data and affords rapid recovery of data. A Split-Mirror snapshot references all of the data on a set of mirrored drives. A Split-Mirror snapshot grabs a snapshot of the entire volume, which simplifies the recovery process. There are, however, some issues inherent in using SAN Snapshots to create “Backups” of databases.

Full recovery using a Copy-on-Write snapshot requires that all previous snapshots be available. Creation of Split-Mirror snapshots tends to be slower than Copy-on-Write snapshots and the storage requirements tend to increase over time. In either case, the storage requirements eventually would become rather onerous since there is no compression. Another consideration is that in order to generate a recoverable snapshot, all of the I/O on the server would have to be stopped. Add to that it’s very likely that the SAN Snapshots will fall to the purview of someone known as a SAN or Storage administrator and not a DBA. (Somehow, I rather doubt that the SAN or Storage Admin will be catching a lot of heat when a critical database goes belly up.) Another important consideration is that each SAN vendor implements SAN Snapshots in a somewhat different manner. In this case a SAN Snapshot was merely an academic discussion given that full snapshots of several volumes, none of which were mirrored, would have been required.

2. Backup and Restore. This is the most readily available approach to backing up and restoring an Analysis Services database as the Backup/Restore functionality is readily available in the product. This functionality can be readily accessed using the GUI in SQL Server Management Studio, using XMLA commands like the following to create a backup:

Adventure Works

C:\PUBLIC\AWDEMO.ABF

with the following XMLA command to restore:

C:\PUBLIC\AWDEMO.ABF

Adventure Works

implemented via the AMO API using code similar to the following to create a backup:

using System;

using System.Collections.Generic;

using System.Linq;

using System.Text;

using Microsoft.AnalysisServices;

namespace BKUP

{

class Program

{

static void Main(string[] args)

{

Server asServer = new Server();

asServer.Connect("localhost");

Database asDB = asServer.Databases.FindByName("Adventure Works");

asDB.Backup("C:\\PUBLIC\\ASDEMO.ABF");

asServer.Disconnect();

asServer.Dispose();

}

and the following AMO code to restore the database

using System;

using System.Collections.Generic;

using System.Linq;

using System.Text;

using Microsoft.AnalysisServices;

namespace RESTORE

{

class Program

{

static void Main(string[] args)

{

Server asServer = new Server();

asServer.Connect("localhost");

asServer.Restore("C:\\PUBLIC\\ASDEMO.ABF","Adventure Works",true);

asServer.Disconnect();

asServer.Dispose();

}

All three are relatively straight forward approaches to backup and restore, using functionality that is built into the product and make this option very appealing. However, there are a few aspects of the native Backup/Restore functionality that make it problematic with extremely large databases. One factor that becomes an extremely important consideration is that the processes for Analysis Services Backup and Restore operations use a limited and fixed number of threads which limits degree of parallelism that can be achieved. The net result is that the native backup/restore operations are roughly comparable in performance to standard file copy operations. While it would be a nice feature to have, Analysis Services doesn’t have functionality similar to the Differential Backups that are available in the SQL Server database engine. In this case, it was known that the database backup was being terminated without completing at 35 hours, so this was obviously not an option. Even if native backup had been an option, we knew that the restore operation would require more than 35 hours making this a non-viable option.

3. Redeploy and fully process. This is obviously one solution, which requires nothing more than having either a copy of the database, in its current state, as a project or the XMLA script to re-create the database. On the positive side, the metadata would be pristine. One slight problem with this approach is that fully processing a multi-terabyte database is typically going to require multiple days, if not weeks, to complete. The amount of time required to fully process a large mission critical database is probably not going to make this an acceptable approach to disaster recovery. In this particular case, fully processing the database would have taken in excess of nine (9) days to complete, so that discussion with the CEO, CFO, and CIO would have been something less than pleasant and cordial.

4. Use Synchronization: Synchronization is another functionality that is natively available in Analysis Services. The product documentation indicates that it can be used to either deploy a database from a staging server to a production server or to synchronize a database on a production server with changes made to the database on a staging server. In any event, the Synchronize functionality does allow an administrator to effectively create and periodically update a copy of an Analysis Services database on another server. This copies the data and metadata from a source server to a destination server. One of the benefits is that the database remains accessible on the destination server, affording users the ability to continue executing queries on the destination server until the synchronization completes and queries are then executed against the newly copied data. Much like Backup/Restore, the Synchronization functionality is readily available in the product and can accessed using the GUI in SQL Server Management Studio or using XMLA commands like the following to the following:

Provider=MSOLAP;Data Source=MyServer;Integrated Security=SSPI

Adventure Works DW 2012 - EE

SkipMembership

true

Using Synchronization for the purpose of creating a “backup” from a production system requires that a destination server that is at least the same service pack level as the Source production server (ideally one would want the identical build number). Using Synchronization also requires that the server being used as the destination have storage capacity equivalent to the source (in this case Production) server. The only server available as a possible destination server was the development server that was being used to make, test, and then push modifications to the design of the database on the production server. For some strange reason, the team doing the database development/modification work had some pretty strong reservations about overwriting the work they were doing in the development environment.

Those were the options that were considered and unfortunately, for one reason or other, none were acceptable. That meant that it was time to start getting creative. I knew that Care Rabeler had done some work with copying databases for a scale out solution, but that was using SAN Snapshots. I was also very aware of a Technet article by Denny Lee and Nicholas Dritsas (https://technet.microsoft.com/library/Cc966449) related to a scale out solution using a SQL Server Integration Services (SSIS) package with Robocopy to copy metadata and data files from multiple databases from a processing server to the data folders of a group of query servers.

Armed with an idea and some information (I know, it’s a dangerous combination) related to using a single instance of the multi-threaded version of Robocopy, it seemed like the beginnings of a pretty tantalizing solution. Rather than copy the entire data directory, all that was really necessary was to copy the data for a single database. The initial plan was to detach the database, use Robocopy to move the data to a “safe” storage location and then re-attach the database. Sounded simple enough, except for a slight complicating factor. The database was nearly three terabytes in size and the data were distributed across eight LUNs.

Detaching and re-attaching an Analysis Services database is a relatively trivial matter that can be easily accomplished from SSMS, but since this was a job that should be scheduled to run on a weekend there was a strong desire to automate the process as much as possible. Building on the prior use of SSIS with Robocopy by Denny Lee and Nicholas Dritsas, it was decided to use an SSIS package to contain and automate the entire process. This had several advantages.

1. This would allow the database to be detached then on success of that operation begin the copy operation

2. Since the data were distributed across eight drives, it would be possible to execute eight instances of Robocopy in parallel (one instance for each drive containing data).

3. Since only the data on each drive was required, it wasn’t necessary to copy the contents of the entire drive which allowed copying a single directory and the subdirectories it contained.

4. Since there were eight LUNs from which data were being copied, it made sense to copy data to eight separate LUNs on a storage server to avoid significant disk I/O contention on the target server.

5. The on completion precedence constraints on the robocopy tasks could be combined with an AND condition so that the database would be re-attached only after all of the data had been copied to a storage location.

A very simple command line utility that could be used to detach or attach a database was really all that was required. Since there wasn't such a utility readily available, it was time to put on the developer hat and start slinging little bit of code. That effort resulted in the following application code:

using System;

using System.Collections.Generic;

using System.Linq;

using System.Text;

using Microsoft.AnalysisServices;

namespace DropAdd

{

class Program

{

static int Main(string[] args)

{

int returnval=0;

switch (args.Count().ToString())

{

case "2":

{

string servername = args[0].ToString().ToUpper().Trim();

string databasename = args[1].ToString().ToUpper().Trim();

ServerApp DetachIt = new ServerApp();

returnval = DetachIt.Detach(servername, databasename);

break;

}

case "3":

{

string servername = args[0].ToString().ToUpper().Trim();

string filepathname = args[1].ToString().ToUpper().Trim();

string databasename = args[2].ToString().ToUpper().Trim();

ServerApp AttachIt = new ServerApp();

returnval = AttachIt.Attach(servername, filepathname, databasename);

break;

}

default:

{

Console.WriteLine("Incorrect number of parameters");

Console.WriteLine("dropadd server_name database_name");

Console.WriteLine("dropadd server_name file_path database_name");

Console.ReadLine();

returnval = 0;

break;

}

return returnval;

}

class ServerApp

{

public int Attach(string ServerName, string FilePathName, string DatabaseName)

{

Server asServer = new Server();

int outcome = 0;

asServer.Connect(ServerName.ToString().Trim());

try

{

Database AsDB = asServer.Databases.FindByName(DatabaseName.ToString().Trim());

if (AsDB != null)

{

outcome = 0;

}

else

{

asServer.Attach(FilePathName.ToString().Trim());

outcome = 1;

}

catch (Exception goof)

{

outcome = 0;

}

finally

{

asServer.Disconnect();

asServer.Dispose();

}

return outcome;

}

public int Detach(string ServerName, string DatabaseName)

{

Server asServer = new Server();

int outcome = 0;

asServer.Connect(ServerName.ToString().Trim());

try

{

Database AsDB = asServer.Databases.FindByName(DatabaseName.ToString().Trim());

if (AsDB != null)

{

AsDB.Detach();

outcome = 1;

}

catch (Exception goof)

{

outcome = 0;

}

finally

{

asServer.Disconnect();

asServer.Dispose();

}

return outcome;

}

Using that code, all that was necessary to detach a database was execute the DropAdd command line utility, passing the Server Name and Database Name as parameters. When it became necessary to attach a database, it was just a matter of executing the DropAdd command line utility passing the Server Name, File path to the database, and Database name as parameters.

Having addressed both detaching and re-attaching the database, it was necessary to consider how to best use Robocopy to move the data from the production server to a storage location. A small scale test using robocopy with the default threading option of 8, worked reasonably well. But since the design of the database distributed data across eight LUNS, it would be necessary to execute robocopy once for each LUN on which data were stored. Running eight instances of robocopy in serial would be a bit time consuming and quite honestly it was suspected that doing so would run well past the 35 hour window for backup creation. An associated problem was determining when the last instance of Robocopy had completed execution. That lead to a decision to execute eight instances of Robocopy in parallel.

The result was the design of an SSIS package looking something like the following:

The SSIS package simply consisted of a set of 10 Execute Process tasks, with the following components:

Detach Database Detach the database from the server

Robocopy Data Dir Copy the data from the database directory

Robocopy G Copy data from the Data Directory on the G drive

Robocopy H Copy data from the Data Directory on the H drive

Robocopy I Copy data from the Data Directory on the I drive

Robocopy J Copy data from the Data Directory on the J drive

Robocopy K Copy data from the Data Directory on the K drive

Robocopy L Copy data from the Data Directory on the L drive

Robocopy M Copy data from the Data Directory on the M drive

Attach Database Attached the database on completion of the copies

For the “Detach Database” task, the following properties were set on the Process tab:

Property	Value
RequiredFullFileName	True
Executable	C:\CustomApps\DroppAdd.exe
Arguments	MyServer “My Big Database”
FailTaskIfReturnCodeIsNotSuccessValue	True
SuccessValue	1
TimeOut	0
WindowStyle	Hidden

Precedence Constraint

Success

In order to ensure that the data could be copied to a “safe” storage location, it is absolutely essential that the database be detached from the server in order to prevent write operations from processing which could result in files on the destination storage location becoming corrupt.

For the “Robocopy Data Dir” task, the following properties were set on the Process tab:

Property	Value
RequiredFullFileName	True
Executable	C:\Windows\System32\Robocopy.exe
Arguments	"C:\Program Files\Microsoft SQL Server\MSAS10_50.MSSQLSERVER\OLAP\Data\My Big Database.17.db" "\\StorageServer\e$\Main" /S /PURGE
FailTaskIfReturnCodeIsNotSuccessValue	False
SuccessValue	1
TimeOut	0
WindowStyle	Hidden

Precedence Constraint

Completion

For the “Robocopy G” task, the following properties were set Process tab:

Property	Value
RequiredFullFileName	True
Executable	C:\Windows\System32\Robocopy.exe
Arguments	"G:\Program Files\Microsoft SQL Server\ OLAP\Data " "\\StorageServer\G$\G_Drive" /S /PURGE
FailTaskIfReturnCodeIsNotSuccessValue	False
SuccessValue	1
TimeOut	0
WindowStyle	Hidden

Precedence Constraint

Completion

For the “Robocopy H” task, the following properties were set on the Process tab:

Property	Value
RequiredFullFileName	True
Executable	C:\Windows\System32\Robocopy.exe
Arguments	"H:\Program Files\Microsoft SQL Server\ OLAP\Data " "\\StorageServer\H$\H_Drive" /S /PURGE
FailTaskIfReturnCodeIsNotSuccessValue	False
SuccessValue	1
TimeOut	0
WindowStyle	Hidden

Precedence Constraint

Completion

For the “Robocopy I” task, the following properties were set on the Process tab:

Property	Value
RequiredFullFileName	True
Executable	C:\Windows\System32\Robocopy.exe
Arguments	"I:\Program Files\Microsoft SQL Server\ OLAP\Data " "\\StorageServer\I$\I_Drive" /S /PURGE
FailTaskIfReturnCodeIsNotSuccessValue	False
SuccessValue	1
TimeOut	0
WindowStyle	Hidden

Precedence Constraint

Completion

For the “Robocopy J” task, the following properties were set on the Process tab:

Property	Value
RequiredFullFileName	True
Executable	C:\Windows\System32\Robocopy.exe
Arguments	"J:\Program Files\Microsoft SQL Server\ OLAP\Data " "\\StorageServer\J$\J_Drive" /S /PURGE
FailTaskIfReturnCodeIsNotSuccessValue	False
SuccessValue	1
TimeOut	0
WindowStyle	Hidden

Precedence Constraint

Completion

For the “Robocopy K” task, the following properties were set on the Process tab:

Property	Value
RequiredFullFileName	True
Executable	C:\Windows\System32\Robocopy.exe
Arguments	"K:\Program Files\Microsoft SQL Server\ OLAP\Data " "\\StorageServer\K$\K_Drive" /S /PURGE
FailTaskIfReturnCodeIsNotSuccessValue	False
SuccessValue	1
TimeOut	0
WindowStyle	Hidden

Precedence Constraint

Completion

For the “Robocopy L” task, the following properties were set on the Process tab:

Property	Value
RequiredFullFileName	True
Executable	C:\Windows\System32\Robocopy.exe
Arguments	"L:\Program Files\Microsoft SQL Server\ OLAP\Data " "\\StorageServer\L$\L_Drive" /S /PURGE
FailTaskIfReturnCodeIsNotSuccessValue	False
SuccessValue	1
TimeOut	0
WindowStyle	Hidden

Precedence Constraint

Completion

For the “Robocopy M” task, the following properties were set on the Process tab:

Property	Value
RequiredFullFileName	True
Executable	C:\Windows\System32\Robocopy.exe
Arguments	"M:\Program Files\Microsoft SQL Server\ OLAP\Data " "\\StorageServer\M$\M_Drive" /S /PURGE
FailTaskIfReturnCodeIsNotSuccessValue	False
SuccessValue	1
TimeOut	0
WindowStyle	Hidden

Precedence Constraint

Completion

For the “Attach Database” task, the following properties were set on the Process tab:

Property	Value
RequiredFullFileName	True
Executable	C:\CustomApps\DroppAdd.exe
Arguments	MyServer "C:\Program Files\Microsoft SQL Server\MSAS10_50.MSSQLSERVER\OLAP\Data\My Big Database.17.db" “My Big Database”
FailTaskIfReturnCodeIsNotSuccessValue	True
SuccessValue	1
TimeOut	0
WindowStyle	Hidden

Precedence Constraint

None

This all assumed that the account that would be executing the job had Administrator permissions for the Analysis Services service, as well as sufficient permissions to access files on each of the drives that contained data. It also required that the account have sufficient permissions to write to the destination drives that were being used to store the files of what would become an uncompressed backup.

It seemed prudent to compare performance with the built-in backup functionality, so being an intrepid soul, I decided to test it out with a version of the Adventure Works database which had been modified to distribute partitions relatively evenly across eight logical drives. The native Backup functionality required 55 seconds to create the backup on an 8 core machine with 24 Gb of RAM. Feeling very confident with the newly minted solution, it was a bit disappointing to find that it took right at 53 seconds to create the backup. However, since Robocopy can be used to copy only changed files, it was decided to process three or four partitions then run the comparison test again. This time, backup again required 55 seconds but the SSIS solution completed in 11 seconds. A good indication that even though a full “backup” of the multi-terabyte database may not be achieved on the first execution there was an extremely good chance that we would have a complete copy following a second execution of the SSIS package.

That meant it was time for the acid test to see how well this solution would perform in the production environment. When the backup window opened, the SSIS package was executed. Approaching the 35 hour mark, the SSIS package had not yet completed execution so it was decided to terminate the package and run the “Attach Database” task to re-attach the database. Somewhat disappointing, but it was encouraging to find that the formerly empty E drive now contained approximately 2.5 Terabytes of data so it was not a total failure. On that basis, it decided to leave the solution in place and allow it to run during the next “backup” window.

When the next backup window opened, the SSIS package began executing, and it was extremely encouraging to find that it completed in seven hours. Checking the E drive, it now contained nearly three terabytes of data. The first thought was “SUCCESS” and now it’s time for a nice cold beer. Of course the second thought was something to the effect of “OK, what happens when one of the disks goes belly up or one of the developers does a full process on one of the dimensions.” Followed by “we have a Backup solution and an uncompressed Backup but no way to restore it.”

Time to go back to work to build another SSIS package that could be used to restore the database. But since we had a “backup” solution, the restore would be simple. It was just a matter of reverse engineer the “backup” solution. This task, however, would be simpler since we would be able to recycle the logic and some of the bits used to create the “backup” solution. It was known that the DropAdd code could be re-used to detach and attach the database. It was also a relatively trivial matter to simply change the order of the parameters passed to the tasks that executed robocopy. Designing a process to restore the database presented a new challenge, in the form of “What happens in the case of a total system failure and it becomes necessary to restore the database to a new but identically configured server?” That would require creating a directory that would contain the database. The result was an SSIS package similar to what you see below:

The "Restore" SSIS package consisted of a set of 1 File Connection Manager, 1 File System Task and 10 Execute Process tasks, with the following components:

MyFileConnection File Connection Manager

Create Database Directory Create the Database Directory if it did not exist

Detach Database Detach the database from the server

Restore Main Data Directory Copy data from backup to the database directory

Restore drive G Data Copy data from backup to the G drive

Restore drive H Data Copy data from backup to the H drive

Restore drive I Data Copy data from backup to the I drive

Restore drive J Data Copy data from backup to the J drive

Restore drive K Data Copy data from backup to the K drive

Restore drive L Data Copy data from backup to the L drive

Restore drive M Data Copy data from backup to the M drive

Attach Database Re-attach the database on completion of the copies

For the “MyFileConnection” File Connection Manager set the following properties on the Process tab:

Property	Value
UsageType	Create Folder
Folder	C:\Program Files\Microsoft SQL Server\MSAS10_50.MSSQLSERVER\OLAP\Data\My Big Database.17.db

For the “Create Database Directory” task, the following properties were set on the Process tab:

Property	Value
UseDirectoryIfExists	True
Name	Create Database Directory
Description	File System Task
Operation	Create Directory
IsSourcePathVariable	False
SourceConnection	MyFileConnection
WindowStyle	Hidden

Precedence Constraint

Success

For the “Detach Database” task, the following properties were set on the Process tab:

Property	Value
RequiredFullFileName	True
Executable	C:\CustomApps\DroppAdd.exe
Arguments	MyServer “My Big Database”
FailTaskIfReturnCodeIsNotSuccessValue	False
SuccessValue	1
TimeOut	0
WindowStyle	Hidden

Precedence Constraint

Success

In this case, it wasn’t really desirable to require success of the detach operation, given that one possible scenario was that the database had never existed on the server and the database was being restored to a “clean” environment.

For the “Restore Main Data Directory” task, the following properties were set on the Process tab:

Property	Value
RequiredFullFileName	True
Executable	C:\Windows\System32\Robocopy.exe
Arguments	"\\StorageServer\e$\Main" "C:\Program Files\Microsoft SQL Server\MSAS10_50.MSSQLSERVER\OLAP\Data\My Big Database.17.db" /S /PURGE
FailTaskIfReturnCodeIsNotSuccessValue	False
SuccessValue	1
TimeOut	0
WindowStyle	Hidden

Precedence Constraint

Completion

For the “Restore drive G Data” task, the following properties were set on the Process tab:

Property	Value
RequiredFullFileName	True
Executable	C:\Windows\System32\Robocopy.exe
Arguments	"\\StorageServer\G$\G_Drive" "G:\Program Files\Microsoft SQL Server\ OLAP\Data " /S /PURGE
FailTaskIfReturnCodeIsNotSuccessValue	False
SuccessValue	1
TimeOut	0
WindowStyle	Hidden

Precedence Constraint

Completion

For the “Restore drive H Data” task, the following properties were set on the Process tab:

Property	Value
RequiredFullFileName	True
Executable	C:\Windows\System32\Robocopy.exe
Arguments	"\\StorageServer\H$\H_Drive" "H:\Program Files\Microsoft SQL Server\ OLAP\Data " /S /PURGE
FailTaskIfReturnCodeIsNotSuccessValue	False
SuccessValue	1
TimeOut	0
WindowStyle	Hidden

Precedence Constraint

Completion

For the “Restore drive I Data” task, the following properties were set on the Process tab:

Property	Value
RequiredFullFileName	True
Executable	C:\Windows\System32\Robocopy.exe
Arguments	"\\StorageServer\I$\I_Drive" "I:\Program Files\Microsoft SQL Server\ OLAP\Data " /S /PURGE
FailTaskIfReturnCodeIsNotSuccessValue	False
SuccessValue	1
TimeOut	0
WindowStyle	Hidden

Precedence Constraint

Completion

For the “Restore drive J data” task, the following properties were set on the Process tab:

Property	Value
RequiredFullFileName	True
Executable	C:\Windows\System32\Robocopy.exe
Arguments	"\\StorageServer\J$\J_Drive" "J:\Program Files\Microsoft SQL Server\ OLAP\Data " /S /PURGE
FailTaskIfReturnCodeIsNotSuccessValue	False
SuccessValue	1
TimeOut	0
WindowStyle	Hidden

Precedence Constraint

Completion

For the “Restore Drive K Data” task, the following properties were set on the Process tab:

Property	Value
RequiredFullFileName	True
Executable	C:\Windows\System32\Robocopy.exe
Arguments	"\\StorageServer\K$\K_Drive" "K:\Program Files\Microsoft SQL Server\ OLAP\Data " /S /PURGE
FailTaskIfReturnCodeIsNotSuccessValue	False
SuccessValue	1
TimeOut	0
WindowStyle	Hidden

Precedence Constraint

Completion

For the “Restore drive L Data” task, the following properties were set on the Process tab:

Property	Value
RequiredFullFileName	True
Executable	C:\Windows\System32\Robocopy.exe
Arguments	"\\StorageServer\L$\L_Drive" "L:\Program Files\Microsoft SQL Server\ OLAP\Data " /S /PURGE
FailTaskIfReturnCodeIsNotSuccessValue	False
SuccessValue	1
TimeOut	0
WindowStyle	Hidden

Precedence Constraint

Completion

For the “Restore drive M Data” task, the following properties were set on the Process tab:

Property	Value
RequiredFullFileName	True
Executable	C:\Windows\System32\Robocopy.exe
Arguments	"\\StorageServer\M$\M_Drive" "M:\Program Files\Microsoft SQL Server\ OLAP\Data " /S /PURGE
FailTaskIfReturnCodeIsNotSuccessValue	False
SuccessValue	1
TimeOut	0
WindowStyle	Hidden

Precedence Constraint

Completion

For the “Attach Database” task, we set the following properties on the Process tab:

Property	Value
RequiredFullFileName	True
Executable	C:\CustomApps\DroppAdd.exe
Arguments	MyServer "C:\Program Files\Microsoft SQL Server\MSAS10_50.MSSQLSERVER\OLAP\Data\My Big Database.17.db" “My Big Database”
FailTaskIfReturnCodeIsNotSuccessValue	True
SuccessValue	1
TimeOut	0
WindowStyle	Hidden

Precedence Constraint

None

For due diligence, it was decided to test the “Restore/Disaster Recovery” package using the same version of Adventure Works that was used for initial testing of the “Backup” package. It was not entirely surprising that copying and attaching the database in that scenario was comparable to restoring from a backup. To test a recovery scenario, it was decided to create a new “backup” using the backup SSIS package, execute a ProcessFull on the Customers dimension and then run the SSIS package to restore the database. It was very encouraging to find that the database was restored to full functionality in roughly 10 seconds.

It didn’t take a long time to receive a painful reminder of the reason that a disaster recovery strategy is important, especially with extremely large databases. Shortly after both the “Backup” and “Restore” SSIS packages were completed, one of the developers on the team managed to accidentally execute a ProcessFull on one of the dimensions used in all of the cubes contained in the database. At this point, there was a choice to be made. Fully processing the database would require a minimum of 9 days, and quite probably longer than that. The “Restore” SSIS package had undergone limited testing but the testing that had been done was extremely encouraging. Ultimately, the “Restore” SSIS package was run and roughly eight (yes, 8) hours later a fully functional multi-terabyte production database was back online.