Microsoft SQL Server Inside Out

It's never too late for Partitioning your Tables

HansPo — Tue, 23 May 2017 15:45:14 GMT

Only a few developers design their tables for partitioning from ground up, simply because they do not envision that a table once might become so big that partitioning would become a real advantage. And even if they do, the fact that partitioning is an Enterprise-only feature (at least until SQL 2016 SP1) keeps them away from using it. Recently, I was involved in a customer case where they did have an audit table where around 10 events were logged per second, all day long. That makes for a little less than 1 Mio. entries per day. And there was the need for a retention period of 30 days. So, once every day they had to delete about 1 Mio. entries from this 30 Mio. row table. This daily delete job was a constant blocker for all other transactions trying to insert into this table during the delete due to the exclusive table lock that was held on the table by the delete job. Not to mention the huge amount of data added to the transaction log with each delete. Since the customer was running on Enterprise Edition, the obvious solution was to partition the table by day. Hence, the daily delete job would be replaced by a job that switches the oldest partition out into an archive table, merging the last partition, and deleting the archive table (since this was on SQL 2014, the new TRUNCATE TABLE WITH PARTITIONS wasn’t yet available). But partitioning the existing table by rebuilding the clustered index would require a maintenance window of several minutes. A price, the customer was not willing to pay. At least, not without evaluating alternatives. So, I came up with a solution that partitions the table within milliseconds and without any major disruption for the transactions running in parallel. You think, that’s impossible? Stay tuned! The crucial point is to dismiss the idea of partitioning the existing data and instead just partition all forthcoming data. I.e. all existing data go into one big partition, whereas all new data go into a new partition that is added every day. It's important that all indexes are already conformant with partition aligned indexes, i.e. the clustered index and all unique indexes must contain the partitioning column as a key column. In our case, the table only had a clustered index on the column that we intended to partition the table with. Hence, all requirements were already met. We started with a partition layout consisting of 3 range right partitions. The left most partition was intentionally left empty, to allow for painless partition merging. The middle partition would store all rows that currently exist in the non-partitioned table. And starting by the next day, all new rows would go into the right most partition. From then on, each new day the customer would expand the partition layout by one new partition until they would have reached the final layout of 30 data partitions. At that point, they would start dropping the oldest partition each day, starting with the big partition we created right at the beginning.

How to code this solution

In the following I’ll outline the steps to be performed for partitioning the table with name AuditTab by the column create_date of type datetime2. Let’s start with creating the partition function and scheme to be used for partitioning our table. First, we must determine the date of the oldest entry in our table. Although, not necessary, we round it down to 00:00 of that day (we want all boundary values of our partition function to be just dates without any time portion). Since, we want the next partition to start the following day, we take this date as the upper boundary. Although, there should not exist any record in our table with a date in the future, we adjust the upper bound just in case (probably it would be better to return an error in that case, but for clarity, we omit error handling here).

DECLARE @start_dt datetime2, @end_dt datetime2;
SELECT @start_dt = MIN(create_date) FROM dbo.AuditTab;
SET @start_dt = DATEADD(dd, DATEDIFF(dd, 0, @start_dt), 0);
SELECT @end_dt = MAX(create_date) FROM dbo.AuditTab;
SET @end_dt = CASE WHEN @end_dt < GETDATE() THEN GETDATE() ELSE @end_dt END;
SET @end_dt = DATEADD(dd, DATEDIFF(dd, 0, @end_dt) + 1, 0);

CREATE PARTITION FUNCTION PF_Date (datetime2) AS RANGE RIGHT FOR VALUES (@start_dt, @end_dt);
CREATE PARTITION SCHEME PS_Date AS PARTITION PF_Date TO ([PRIMARY], [PRIMARY], [PRIMARY]);

Our AuditTab table has been created in the PRIMARY filegroup. It’s important that the middle partition is mapped to the same filegroup where the original table was created in. Next, we create a check constraint on our table enforcing that all current rows (plus those being added until we start partitioning the table) are within the boundaries of the middle partition. This is the most expensive operation in the whole process. Although, it will take only a few seconds on a table of this size, the schema modification lock, which is being acquired while creating this check constraint, blocks all DML operations on the table. Therefore, we perform this operation right ahead (preferably at a period of lower insert activity). DECLARE @check nvarchar(max), @sql nvarchar(max);

SET @check = N' CHECK (create_date >= ''' + CONVERT(nvarchar, @start_dt, 112) +
N''' AND create_date < ''' + CONVERT(nvarchar, @end_dt, 112) + N''')';

SET @sql = N'ALTER TABLE dbo.AuditTab ADD CONSTRAINT chk_date' + @check;
EXEC sp_executesql @sql;

Now, we create a temporary table that has an identical structure as our original table, is in the same filegroup, has the same indexes, and the same constraints. This is necessary, because later we want to switch the data from our original table into this temp table. Thereby, we can partition the empty original table, and later switch the data back from the temp table. This has the advantage that we do not alter the underlying database object, i.e. object ownership, all permission settings, and other constructs that reference the table, like triggers or extended properties, etc. remain unaffected.

CREATE TABLE dbo.AuditTab_Temp (…,create_date datetime2 NOT NULL, …) ON [PRIMARY];
CREATE CLUSTERED INDEX ix ON dbo.AuditTab_Temp (create_date);

SET @sql = N'ALTER TABLE dbo.AuditTab_Temp ADD' + @check;
EXEC sp_executesql @sql;

Now, we perform the actual transformation of the original table into a partitioned table. All operations are grouped within a transaction, so if anything fails, the transaction would be rolled back. First, we switch the original table into the temporary table we just created. Then, we partition the, now empty, original table by rebuilding the clustered index. Finally, we switch the temp table back into the middle partition of the now partitioned original table. Since we did create the check constraint that is necessary for this switch already on the original table and then on the temporary table, it is not necessary to perform this expensive check constraint creation on the temporary table within the transaction after switching the data from the original table into the temporary table.

BEGIN TRAN;
ALTER TABLE dbo.AuditTab SWITCH TO dbo.AuditTab_Temp;
CREATE CLUSTERED INDEX ix_AuditTab ON dbo.AuditTab (create_date) WITH (DROP_EXISTING = ON) ON PS_Date (create_date);
ALTER TABLE dbo.AuditTab_Temp SWITCH TO dbo.AuditTab PARTITION 2;
COMMIT TRAN;

Note, this transaction only consists of meta-data changes and therefore performs very fast. In principal, nothing has the potential to fail within this transaction since we enforce that all data fit into the target partition by means of the check constraint. Anyway, it’s recommended to put this transaction within a try-block, and rollback the transaction in the catch-block. Finally, we drop the additional objects being created, i.e. the temp table and the check constraint.

DROP TABLE dbo.AuditTab_Temp;
ALTER TABLE dbo.AuditTab DROP CONSTRAINT chk_date;
ALTER TABLE dbo.AuditTab SET (LOCK_ESCALATION = AUTO);

We also set the LOCK_ESCALATION setting for our original table to AUTO, which causes lock escalations on the partition level, instead of the table level. Thereby, if saving space is a must, we may continue deleting the data that is older than 30 days from the big partition on a daily basis, but now without having to lock the whole table. Hence, the deletes do not affect the partitions where the new inserts are occurring and hence, increasing the concurrency.

Defining the partition management job

Now that the table is being partitioned, the only thing left is to define the daily partition management activities. We create a SQLAgent job for this, which runs every day before midnight, starting at the day where we partitioned the table. This job consists of one TSQL step executing the following batch: DECLARE @fid int, @num_part int, @first_val sql_variant;

SELECT @fid = function_id, @num_part = fanout
FROM   sys.partition_functions WHERE name = 'PF_Date';

IF @num_part >= 32
BEGIN
CREATE TABLE dbo.AuditTab_Archive (…, create_date datetime2 not null, …);
CREATE CLUSTERED INDEX ix ON dbo.AuditTab_Archive (create_date);
ALTER TABLE dbo.AuditTab SWITCH PARTITION 2 TO dbo.AuditTab_Archive;
DROP TABLE dbo.AuditTab_Archive;

SELECT @first_val = value FROM sys.partition_range_values
WHERE  function_id = @fid AND boundary_id = 1 AND parameter_id = 1;

ALTER PARTITION FUNCTION PF_Date() MERGE RANGE (@first_val);
END

ALTER PARTITION SCHEME PS_Date NEXT USED [PRIMARY];
ALTER PARTITION FUNCTION PF_Date() SPLIT RANGE (DATEADD(dd, DATEDIFF(dd, 0, GETDATE()) + 2, 0));

As long as we haven’t reached the final target of 32 partitions (note there is one empty partition at the beginning and one at the end), we just add a new partition, mapped to the PRIMARY filegroup, by splitting the last partition on the date that follows the next day. Hence, starting at midnight all new records will go into the partition preceding the new one (the last partition is always left empty so that the SPLIT operation is always performed on an empty partition). Once we have reached 30 partitions with data, we drop the partition with the oldest data, which always happens to be partition 2, by switching the data out into an archive table (which later is being dropped), and merging the (empty) partitions 1 and 2 using the lowest boundary value in our partition range.

Conclusion

In case you intend to partition a table where it is acceptable not to partition the existing data, but only all future data, using the strategy outlined above it is possible to partition even huge tables within seconds without major disruption for parallel workload. By the way, this strategy also has been implemented in the new version of PTF (see https://ptf.codeplex.com/) by setting the parameter @fast_mode = 1 in procedure PartitionTable. Hence, in PTF executing the following command would perform the identical steps: EXEC ptf.PartitionTable 'dbo.AuditTab', 'PS_Date', 'create_date', @fast_mode = 1

Partitioning of Temporal Tables in SQL Server 2016

HansPo — Wed, 20 Jan 2016 05:51:09 GMT

Microsoft SQL Server 2016 introduces system-versioned temporal tables, which allow for automatically keeping a history of data changes in a separate history table and to retrieve the status of your data at any time in the past. Here I don’t want to go into all the gory details of their implementation. See the relevant BOL pages https://msdn.microsoft.com/en-US/library/dn935015.aspx or https://blogs.technet.com/b/dataplatforminsider/archive/2016/01/11/effortlessly-analyze-data-history-using-temporal-tables.aspx for an exhaustive description of temporal tables.

In this Blog I just want to show how table partitioning affects temporal tables. Like any other table, temporal tables can be partitioned (at least, if you are running on Enterprise Edition of SQL Server). In order to partition a temporal table while creating the table, simply add the ON () clause to the CREATE TABLE statement, e.g.

CREATE TABLE MyTemporalTable (

Id int PRIMARY KEY,

Data nvarchar(1000) NULL,

ValidFrom datetime2 GENERATED ALWAYS AS ROW START,

ValidTo datetime2 GENERATED ALWAYS AS ROW END,

PERIOD FOR SYSTEM_TIME (ValidFrom, ValidTo)

) ON MyPartScheme (Id)

WITH (SYSTEM_VERSIONING = ON)

But note, when doing so, the partitioning column has to be part of the primary key that has to be specified with each temporal table. If you do not explicitly specify a name for the history table, one will be generated automatically for you with the name MSSQL_TemporalHistoryFor_nnnn, where nnnn is the object ID of the temporal table. And, this history table is going to be partitioned the same way as the underlying temporal table, i.e. using the same partitioning column and the same partition scheme and function. But, having more than one table partitioned using the same partition function is not a good practice, since every Merge or Split operation on the partition function affects both tables at the same time and may cause undesired data movements if you do not pay attention to it. Therefore, I strongly advise not to partition a temporal table while creating it together with the associated history table. And, by the way, I do not see any good reason for partitioning the temporal table and its history table on the very same column and using the same partition scheme and function, since both tables serve different purposes.

Partitioning the Temporal Table

So let us view the aspects of partitioning for each of the tables separately. For the temporal table you typically might consider partitioning if you want to balance data storage on different filegroups and their associated physical data files, or if you want to utilize partitions for fast data loading. Note, that adding new rows to a temporal table has no effect on the history table. Therefore, switching data into a partition of a temporal table is a valid operation. For a table to be switched in the PERIOD FOR SYSTEM_TIME columns must be defined identically, except for the HIDDEN attribute, and the indexes must match.

If the temporal table is being created like:

CREATE TABLE MyTemporalTable (

Id int CONSTRAINT PK_MyTemporalTable PRIMARY KEY,

Data nvarchar(1000) NULL,

ValidFrom datetime2 GENERATED ALWAYS AS ROW START,

ValidTo datetime2 GENERATED ALWAYS AS ROW END,

PERIOD FOR SYSTEM_TIME (ValidFrom, ValidTo)

)

WITH (SYSTEM_VERSIONING = ON)

You may partition this table by recreating the primary key:

CREATE UNIQUE CLUSTERED INDEX PK_MyTemporalTable ON MyTemporalTable (Id)

WITH (DROP_EXISTING = ON) ON MyPartScheme (Id)

Next, create the staging table to be switched in and fill it with data (declaring the system_time columns as HIDDEN might be beneficial here):

CREATE TABLE MyTemporalStaging (

Id int PRIMARY KEY,

Data nvarchar(1000) NULL,

ValidFrom datetime2 GENERATED ALWAYS AS ROW START HIDDEN,

ValidTo datetime2 GENERATED ALWAYS AS ROW END HIDDEN,

PERIOD FOR SYSTEM_TIME (ValidFrom, ValidTo)

)

And finally, after splitting the last partition, switch the staging table into the new last partition of the temporal table:

ALTER TABLE MyTemporalStaging SWITCH TO MyTemporalTable PARTTION @last_partition

On the other hand, switching partitions out from the temporal table effectively means deleting rows and therefore must be reflected in the history table as well. Because this cannot be handled by the system, switching partitions out of a temporal table is not a supported operation and returns error 13546 when doing so. If you really need to switch out partitions from a temporal table, you first have to disable system versioning on the temporal table, and turn it on again after the switch.

Partitioning of the History Table for Managing Data Retention

A temporal history table steadily grows as you perform data modifications on your temporal table. Hence, the history table might become really big, and at some point you may face the need to remove old and no longer needed row versions from your history table in order to reclaim space. As of CTP 3 automatic data retention isn’t yet implemented for temporal history tables. So you have to do it on your own. See also https://msdn.microsoft.com/en-us/library/mt637341.aspx.

But, simply deleting rows from the history table isn’t allowed. You have to disable system versioning prior to this, and you better perform these steps within a transaction in order to block any data modification on the temporal table while system versioning is being disabled. So, this would look like:

BEGIN TRANSACTION

ALTER TABLE SET (SYSTEM_VERSIONING = OFF)

DELETE FROM WHERE <= @cut_off_date

ALTER TABLE SET (SYSTEM_VERSIONING = ON

(HISTORY_TABLE = , DATA_CONSISTENCY_CHECK = OFF))

COMMIT TRANSACTION

Depending on the amount of rows being deleted, this might cause lock escalation and result in an exclusive table lock on the history table, which is further propagated to the temporal table, and thus, also blocks all read operations on the temporal table for the duration of this transaction. Moreover, this operation may result in a huge amount of rows being generated in the transaction log. Thus, for a big history table this is not the recommended procedure for data cleansing.

Since switching partitions out of the history table is a supported operation, partitioning of the history table is the preferred method to implement such a data retention scheme on a temporal history table. Simply create a partition function that covers your envisioned retention period and deletion frequency and partition the history table on the end column of the system time period. Note, that the clustered index that is created automatically on the history table has the two system time period columns as key columns and therefore, is perfectly suited for partitioning on this column.

If your temporal table has been defined as

CREATE TABLE dbo.TemporalTable (

Id int PRIMARY KEY,

Data nvarchar(1000) NULL,

ValidFrom datetime2 GENERATED ALWAYS AS ROW START,

ValidTo datetime2 GENERATED ALWAYS AS ROW END,

PERIOD FOR SYSTEM_TIME (ValidFrom, ValidTo)

)

WITH (SYSTEM_VERSIONING = ON (HISTORY_TABLE = dbo.TemporalHistory))

the history table, if it does not yet exist, will be created equivalent to

CREATE TABLE dbo.TemporalHistory (

Id int NOT NULL,

Data nvarchar(1000) NULL,

ValidFrom datetime2 NOT NULL,

ValidTo datetime2 NOT NULL,

INDEX ix_TemporalHistory CLUSTERED (ValidTo, ValidFrom) WITH (DATA_COMPRESSION = PAGE)

)

Therefore, if you’re going to partition the history table on column ‘ValidTo’ using the previously created partition scheme ‘MyPartScheme’, you can accomplish this by recreating the clustered index:

CREATE CLUSTERED INDEX ix_TemporalHistory ON dbo.TemporalHistory (ValidTo, ValidFrom)

WITH (DATA_COMPRESSION = PAGE, ONLINE = ON, DROP_EXISTING = ON) ON MyPartScheme (ValidTo)

If you have created your own indexes on the history table to replace or supplement the default clustered index, the procedure to partition the history table might look slightly different. Especially, if you have defined a columnstore index, that index has to be dropped prior to partitioning the history table, and recreated afterwards.

Now, whenever you are going to delete the data in the partition with the oldest ‘ValidTo’ values, you have to create an archive table that matches the structure and the clustered index of the history table, switch the partition out to this archive table, merge the two left most partitions, and finally drop the archive table. Let’s say ‘MyPartFunction’ is the name of the partition function our partition scheme ‘MyPartScheme’ is based on, then the following batch would accomplish this maintenance job (with RANGE RIGHT partitions, the leftmost partition should be left empty):

DECLARE @v datetime2, @p int;

SELECT @v = CONVERT(datetime2, prv.value), @p = 1 + pf.boundary_value_on_right

FROM sys.partition_functions pf JOIN sys.partition_range_values prv

ON prv.function_id = pf.function_id AND prv.parameter_id = 1 AND prv.boundary_id = 1

WHERE pf.name = 'MyPartFunction';

CREATE TABLE dbo.TemporalHistoryArchive (

Id int NOT NULL,

Data nvarchar(1000) NULL,

ValidFrom datetime2 NOT NULL,

ValidTo datetime2 NOT NULL

INDEX ix_TemporalHistoryArchive CLUSTERED (ValidTo, ValidFrom) WITH (DATA_COMPRESSION = PAGE)

);

ALTER TABLE dbo.TemporalHistory SWITCH PARTITION @p TO dbo.TemporalHistoryArchive;

ALTER PARTITION FUNCTION MyPartFunction() MERGE RANGE (@v);

DROP TABLE dbo.TemporalHistoryArchive;

Although SQL Server 2016 added the WITH PARTTITIONS clause to the TRUNCATE TABLE statement to allow for operations like that, using the statement “TRUNCATE TABLE TemporalHistory WITH (PARTITIONS (@p))” returns error 13545. Hence, even if you do not intend to keep the archived data, you have to follow the more cumbersome process of creating an archive table, and dropping it again after having switched out the partition to it.

Note, that the next version of PTF on https://ptf.codeplex.com will support SQL Server 2016 and temporal tables. Thereby, the above maintenance script could be substituted with the single statement:
EXEC ptf.DropPartitions 'dboTemporalHistory'

Summary

With temporal tables, both the temporal table itself, as well as its associated history table can be partitioned. On the partitioned temporal table, the only supported operation is the switch in, while on the partitioned history table we only may switch out partitions without disabling system versioning.

Especially, partitioning the history table is recommended if you want to implement some means of data cleansing in response to excessive data growth in this table.

Partitioned Table Framework (PTF) Project has been published on CodePlex

HansPo — Fri, 09 Nov 2012 07:35:23 GMT

Today I did publish the PTF project on CodePlex (see https://ptf.codeplex.com/) which consists of a set of T-SQL stored procedures for managing partitioned tables in SQL Server. These procedures may be used in your SQL Agent Jobs or SSIS SQL Tasks that load data into staging tables of your data warehouse and then switch these tables into partitions of your partitioned fact tables, i.e. implementing a sliding window scenario. All SQL Server versions and Editions that allow for partitioned tables are being supported.

The following activities can be accomplished:

Creating a staging table.
Preparing a staging table, i.e. creating Indexes and constraints.
Performing partition switches and Partition merges and splits.
Unloading or uploading of specific partitions.
Converting an unpartitioned table into a partitioned one.

For detailed documentation download the PTF.docx file from the codeplex project.

Hardening an ASP.NET session state database

HansPo — Mon, 10 Jan 2011 02:28:29 GMT

Recently I did come across a problem with the ASPState database of ASP.NET. This database is used to store the ASP.NET user session data in a SQL Server instance. The database is being installed by executing Aspnet_regsql.exe with parameter –ssadd (see Session-State Modes). Principally there are two configuration modes: persistent and non-persistent, the latter being the default. In persistent mode the two tables ASPStateTempSessions and ASPStateTempApplications, which hold the current ASP state, are being created in the ASPState database, whereas in the non-persistent mode both tables are being created in tempdb and are recreated each time SQL Server restarts using the startup procedure CreateTempTables. Thus, in persistent mode the ASP state can survive a SQL Server restart, while in non-persistent mode the state is lost whenever SQL Server restarts, which usually is not a problem.

This setup works perfectly for members of the sysadmin fixed server role or the owner of the ASPState database. But, since no permissions are being granted by the setup program, all other principals do not have access to the ASP.NET procedures. When ASP.NET is configured to store its state in SQL Server, the user that connects to the ASPState database is hardcoded together with the password in the connection string stored within the element of the Web.config file. Although there are ways to encrypt this information (see Encrypting Configuration Information Using Protected Configuration), this still could a pose the risk for a security breach when using an administrative account for this purpose. Thus, in order to harden your ASP.NET environment it would be more desirable to have a low privileged SQL Server login and granting this login access to the ASPState database. Now, the only thing that is left is to grant this database user (or the public role) EXECUTE rights on the ASPState procedures. That is all that needs to be done for the persistent mode. But, for the non-persistent mode things are a little bit more tricky.

In the persistent case, it is sufficient to grant EXECUTE rights on the procedures, because access to the tables works by means of an ownership chain, i.e. if the owner of a procedure also owns the tables being accessed by the procedures, no additional checks are being performed when accessing those tables. In the ASPState database the owner of all objects is also the database owner, i.e. the account who did create the ASPState database when running the Aspnet_regsql tool. In the non-persistent mode, where the tables do not reside in the ASPState database but are located in tempdb, there is no direct ownership chain and therefore one more step has to be performed: a cross-database ownership chain has to be established.

First, we need to allow cross-database ownership chains for the participating databases. This can be set server wide by means of a configuration parameter. But this could create undesired security holes. Therefore, it’s always recommended to enable this on the database level. On tempdb this already has been set by default. But on ASPState we have to perform this manually, executing the “ALTER DATABASE ASPState SET DB_CHAINING ON” statement. But still this does not suffice. We are still receiving the error “The INSERT permission was denied on the object 'ASPStateTempSessions', database 'tempdb', schema 'dbo'.” when calling an ASPState procedure, like TempInsertStateItemShort. The problem we are still facing is the fact that cross-database ownership chaining does not work on a user basis, but on a login basis. The owner of the ASPState database is the login that did create the database. But for tempdb it is always the sa login. Thus, for the cross-database ownership chain to work the owners of both databases have to match the same principal. Since it is not practical to change the ownership of tempdb, we have to change it on database ASPState. This is being accomplished by executing “ALTER AUTHORIZATION ON DATABASE::ASPState TO sa”.

To summarize, here are the steps to be performed in order to harden your ASP.NET applications that are using SQL Server mode for storing the session state:

Either create a low privileged login for the identity your ASP.NET applications run with (see Configuring ASP.NET Process Identity) and Integrated Security is used in the sqlConnectionString attribute, or create a SQL Server login to be used for this purpose. After having created the ASPState database by running Aspnet_regsql.exe, define this login as a user on the ASPState database. Then run the following script:

USE master

IF EXISTS (

SELECT * FROM tempdb.sys.tables

WHERE schema_id = 1 AND name = 'ASPStateTempSessions')

BEGIN

ALTER DATABASE ASPState SET DB_CHAINING ON;

ALTER AUTHORIZATION ON DATABASE::ASPState TO sa;

END

USE ASPState

DECLARE @cmd nvarchar(max)

SET @cmd = N''

SELECT @cmd = @cmd +

N'GRANT EXECUTE ON dbo.' + QUOTENAME(name) + ' TO PUBLIC;'

FROM sys.objects

WHERE type = 'P' AND schema_id = 1 AND name LIKE 'Temp%'

EXEC (@cmd)

New Limit for Number of Partitions in SQL Server 2008 SP2

HansPo — Mon, 29 Nov 2010 06:34:09 GMT

Good news for all that suffered from the 1.000 partition limit inherent with the current table partitioning implementation in Microsoft SQL Server 2005 and 2008. With the release of SP2 for SQL Server 2008 the bar has been raised to a new threshold of 15.000 partitions, which should be enough for daily data loads and makes even smaller load cycles a reality. See the whitepaper Support for 15,000 Partitions in SQL Server 2008 SP2 for details.

But this feature does not come out of the box. It has to be enabled on a database by database basis after having upgraded your SQL Server 2008 instances to SP2 (users of SQL Server 2008 R2 still have to wait for SP1 to benefit from this change). You do this by calling the new system stored procedure sp_db_increased_partitions. For instance, to enable the database MyDataWarehouse for increased partitions, simply call:
EXEC sp_db_increased_partitions @dbname = ‘MyDataWarehouse’, @increased_partitions = ‘on’

Note, that this operation increases the version of the database from 655 to 662. This has implications for the ability to upgrade this database or to attach a detached copy of the altered database to another server. The target system must have the same or a higher database version. Thus, you cannot upgrade this server to SQL Server 2008 R2 RTM (which has version 661), or attach the database to a SQL Server 2008 SP1 instance. And there are also implications for log shipping or mirroring where the partner servers also have to be on SP2 level. While with log shipping it is sufficient to run the procedure just on the primary server (the corresponding log entry is propagated to the secondary servers and also applied there), with mirroring the mirror has to be broken, and the procedure being executed on both principal and mirror server before the mirror can be re-established.

And there are a few more caveats to consider:

· it can only be applied to user databases.

· you cannot execute this proc within an open transaction.

· you must be the only user in the database since this proc tries to aquire an exclusive lock on the DB.

· this database must not participate in replication, CDC or change tracking.

· it is not supported for this upgraded database to have a nonaligned index defined on any partitioned table.

Also, the partitioning Wizards in SQL Server Management Studio 2008 have not been updated to cater for the increased partition limit. Therefore, you have to use T-SQL to manage those partitions.

Inside of Table and Index Partitioning in Microsoft SQL Server

HansPo — Fri, 21 Aug 2009 03:23:00 GMT

Introduction

With those huge fact tables, you typically find in today’s data warehouses, you’re often confronted with restrictions concerning the manageability of these tables and their indexes. Some actions like generating full database backups, checking the database consistency, or reorganizing indexes are becoming prohibitive in terms of time required to perform these activities. Additionally, bulk loading big chunks of data into these tables incurs big overhead for index maintenance and transaction logging and results in reduced concurrency on the tables due to blocking.

Most of these problems are being addressed by a feature available in the Enterprise Edition of SQL Server 2005 and later, called table and index partitioning. Quite a few information about partitioning is available in various sources, like SQL Server Books Online, or the excellent White Papers “Partitioned Tables and Indexes in SQL Server 2005" https://msdn.microsoft.com/library/ms345146.aspx, or “Partitioned Table and Index Strategies Using SQL Server 2008" https://msdn.microsoft.com/library/dd578580.aspx. In the following, I’d like to dig a little bit deeper into this topic, showing the effects of partitioning on SQL Server metadata, with special focus on sliding window scenarios. Also, I’ll present some T-SQL queries that might come handy for administering your partitions. And finally, I will talk about the improvements SQL Server 2008 has up its sleeves for partitioning.

Table Partitioning

Partitioning a table means splitting the table data horizontally into smaller units, called partitions, that are easier to manage than the table as a whole. While a non-partitioned table is stored on a single filegroup, each of the partitions of a partitioned table can be mapped to a different filegroup, thus allowing for better control over the physical distribution of table space. With this separation, filegroup backups and checking the consistency of a single filegroup instead of the whole database are becoming a reality. Additionally, by marking some filegroups, that are associated with partitions that do not change, as READ_ONLY, you can take advantage of the new partial backup feature or the reduced locking when querying data on those filegroups.

So how does SQL Server determine, which partition a row of a partitioned table belongs to? For this, we first have to identify the column by which the table is being partitioned. The data type of this column defines the maximum range of possible values, e.g. for integer the range is from negative infinity to positive infinity. Next, this range is further divided into consecutive, non-overlapping sub-ranges, each of them defining a partition range. Thus, the sub-range, into which the value of the partitioning column for a particular row falls, determines the partition the row belongs to.

Therefore, the first decision to make, when creating a partitioned table, is which column to use to partition the table. This column is called the partitioning column. In SQL Server 2005 and SQL Server 2008 only one column can be used for partitioning a table, but this may change in future releases. The decision which column to choose will be based on data distribution and the intended usage of the partitioned table and is mostly based on the same criteria we apply, when identifying the columns of a clustered index. In fact, the partitioning column also has to be a member of the clustered index keys. It’s obvious, that a partitioning column needs to be a fairly static one, as a change of the partitioning column also might result in a change of the partition the row belongs to and thus, the row would have to be copied physically from one partition to another. If the partitioned table is intended for business analysis purpose, the most predominant dimension used for analysis is a good candidate for the partitioning column, e.g. date, product group, sales region. If the table stores historical data and at any time we only want to keep a certain range of data, i.e. we constantly age old data out and move new data in, the table needs to be partitioned on some date value stored in the table. This is the typical usage of the sliding window scenario as we will discover later.

Partition Functions

Once the partitioning column has been identified, we need to define the number of partitions to use and the boundary values for each of the partition ranges. This designation has to be set up by means of a Partition Function. The partition function is being created using the following T-SQL statement:

CREATE PARTITION FUNCTION <name> (<type>) AS RANGE {LEFT | RIGHT} FOR VALUES (v₁, …, v_n);

By specifying n boundary values v₁ through v_n of the given type, this function defines n+1 partitions. A maximum of 1000 partitions can be created this way. The type the function is based on must be of an integral SQL type, LOB’s, xml, timestamp and user defined types are excluded. The order of the boundary values does not matter as long as all values are distinct. The system orders them implicitly in case they are not given in ascending order. LEFT or RIGHT specifies to which side of a boundary value the pertaining partition extends. If the boundary values are arranged in ascending order on a horizontal axis, the partition range alignment can be illustrated as:

             partition_i                         partition_i+1
LEFT:   ---|[---------|]---        RIGHT: ---[|---------]|---
          v_i-1         v_i                       v_i         v_i+1

This arrangement probably becomes more obvious when listing for the partitioning column x the partitions and their pertaining ranges in tabular format:

	RANGE LEFT	RANGE RIGHT
partition₁:	x <= v₁	x < v₁
partition₂:	x > v₁ AND x <= v₂	x >= v₁ AND x < v₂
:
partition_n:	x > v_n-1 AND x <= v_n	x >= v_n-1 AND x < v_n
partition_n+1:	x > v_n	x >= v_n

The choice whether we specify a LEFT or a RIGHT range ordering is basically an arbitrary one with only minor functional differences. But both have their pros and cons. In most scenarios a RIGHT range seems to be easier to formulate. On the other hand, a LEFT range function might be somewhat more intuitive to use when it comes to switching partitions in and out. We will come back to this topic later when talking about sliding window scenarios.

Also keep in mind that partition functions are not static. You can adjust the number of partitions at any time. By merging two partitions into one, you decrease the number by one, and by splitting one partition into two, you increase the number by one.

Partition Schemes

Next, before starting to fill the partitions with data, we have to define where to store the partition data. This is being accomplished by means of a Partition Scheme. Each partition scheme is associated with a specific partition function, and for each partition defined by this function the filegroup where to store the data for this partition is being specified. A partition scheme is created using the following T-SQL syntax:

CREATE PARTITION SCHEME <name> AS PARTITION <partition function> TO (<fg₁>, …, <fg_m>);

There has to be an assignment to a filegroup for each partition defined by the function, and the assignments have to be in order, i.e. partition₁ maps to fg₁, and so on. But it’s also legitimate to specify more filegroups than partitions exist. In this case, the first filegroup in excess will be marked as NEXT USED (more on this later). All filegroups listed must already exist in the database. But the filegroups do not need to be distinct, i.e. several partitions can be mapped to the same filegroup. There’s even a special abbreviated syntax for the case where all partitions map to the same filegroup, ALL TO (<fg>).

Creating Partitioned Tables

Now that we have everything in place, let’s create the partitioned table. There is no big difference from creating a non-partitioned table, except the ON clause at the end of the statement. While for a non-partitioned table we can specify a filegroup where the table should be created on (if nothing is specified, the table is being created on the default filegroup), for a partitioned table we simply specify the partition scheme together with the partitioning column. So, the T-SQL statement for creating a partitioned table looks like:

CREATE TABLE <name> ( … ) ON <partition scheme> (<col>)

The partitioning column specified must be a column of the table (even computed columns are allowed as long as they are persisted) and it must be of the same type (or implicitly convertible to this type) defined for the partition function the partition scheme is associated with. Practically, this column is being defined as NOT NULL, although this is not a requirement. The same partition scheme can be used for other tables as well.

Partitioning Metadata

Now let’s have a closer look at the metadata associated with the various partition objects. SQL Server provides several system catalog views to make the internals of partitioning visible. For a comprehensive description of these catalog views please consult BOL. Here we are just focusing on the most meaningful columns of those views while following the creation process of a sample partition.

So let’s start with creating a partition function for an integer column:

CREATE PARTITION FUNCTION PFTest (int) AS RANGE RIGHT FOR VALUES (10, 20, 30, 40)

Every partition function has an entry in the sys.partition_functions catalog view. The above sample partition function might create the following row:

select name, function_id, fanout, boundary_value_on_right
from sys.partition_functions
where name = N'PFTest'

name function_id fanout boundary_value_on_right
--------- ----------- ----------- -----------------------
PFTest 65546 5 1

The function_id is a unique identifier assigned to each partition function. The fanout column specifies the number of partitions defined by this function (Note: this number is always the number of boundary values given in the partition function definition plus 1). And the value for boundary_value_on_right is 1 for RIGHT aligned partition ranges and is 0 for LEFT ones.

The data type the partition function is based on can be determined from the catalog view sys.partition_parameters:

select parameter_id, TYPE_NAME(system_type_id) as [type]
from sys.partition_parameters
where function_id = 65546

parameter_id type
------------ ----------
1 int

Note that the partitioning catalog views are already prepared to deal with more than one partitioning column. But the current T-SQL syntax only allows to create partitions based on a single partitioning column. Therefore, sys.partition_parameters currently will only have one row per partition function and parameter_id always will be 1.

And finally, the sys.partition_range_values catalog view has an entry for every boundary value given in the function definition:

select boundary_id, parameter_id, value
from sys.partition_range_values
where function_id = 65546

boundary_id parameter_id value
----------- ------------ -----
1           1            10
2           1            20
3           1            30
4           1            40

The boundary values specified in the CREATE PARTITION FUNCTION statement are assigned to boundary_id’s in ascending order. Also note that the value column is of type sql_variant having the base type of the partition function parameter, i.e. SQL_VARIANT_PROPERTY(value, 'BaseType') should give the same type returned by the previous query on sys.partition_parameters.

Before creating the partition scheme, we have to create the filegroups to hold each partition, in case they do not yet exist. It’s also legitimate to place all partitions in the same filegroup, but doing so would sacrifice much of the benefits partitioning does offer to you. For our sample partition scheme let’s create five filegroups named FG1, FG2, FG3, FG4 and FG5 using the ALTER DATABASE ADD FILEGROUP statement.

Now we are ready to create the partition scheme based on our partition function and map each of the 5 partitions to one of our filegroups:

CREATE PARTITION SCHEME PSTest AS PARTITION PFTest TO (FG1, FG2, FG3, FG4, FG5)

This creates an entry for the catalog view sys.partition_schemes that can be queried like:

select name, data_space_id, function_id
from sys.partition_schemes
where name = N'PSTest'

name data_space_id function_id
--------- ------------- -----------
PSTest 65608 65546

The data_space_id uniquely identifies the data space that is associated with the partition scheme (a data space represents the storage scheme of a table/index and either can be a filegroup, or a partition scheme). And function_id links the partition scheme with the partition function it is based on.

The data space of a partition scheme is further divided into destination data spaces where each destination data space is having an entry in the system catalog view sys.destination_data_spaces. Every destination data space is linked to its parent partition scheme through its partition_scheme_id column. The data_space_id column links the destination data space to its pertaining filegroup. The destination_id is equivalent to the partition number the destination data space is mapped to. The following query returns each destination data space for our sample partition scheme and the filegroup assigned to it:

select dds.destination_id, fg.name as [filegroup]
from sys.destination_data_spaces dds join sys.filegroups fg
on dds.data_space_id = fg.data_space_id
where dds.partition_scheme_id = 65608

destination_id filegroup
-------------- ---------
1              FG1
2              FG2
3              FG3
4              FG4
5              FG5

Putting it all together

In order to display all this information within a single query, I have created the following view that will be used extensively within the next sections:

CREATE VIEW [dbo].[PartitionRanges] AS
select pf.name as [partition_function],
       ps.name as [partition_scheme],
       1 as [partition_number],
       case when prv.value is null then NULL else '<' end as [relation],
       prv.value as [boundary_value],
       type_name(pp.system_type_id) as [type],
       fg.name as [filegroup],
       case when ps.name is null then NULL else N'IN USE' end as [status]
from sys.partition_functions pf
join sys.partition_parameters pp on pp.function_id = pf.function_id
left join sys.partition_schemes ps on ps.function_id = pf.function_id
left join sys.destination_data_spaces dds
    on dds.partition_scheme_id = ps.data_space_id and dds.destination_id = 1
left join sys.filegroups fg on fg.data_space_id = dds.data_space_id
left join sys.partition_range_values prv
    on prv.function_id = pf.function_id and prv.parameter_id = 1 and
       prv.boundary_id = 1
where pf.boundary_value_on_right = 1
union all
select pf.name as [partition_function],
       ps.name as [partition_scheme],
       prv.boundary_id + cast(pf.boundary_value_on_right as int) as [partition_number],
       case when pf.boundary_value_on_right = 0 then '<=' else '>=' end as [relation],
       prv.value as [boundary_value],
       type_name(pp.system_type_id) as [type],
       fg.name as [filegroup],
       case when ps.name is null then NULL else N'IN USE' end as [status]
from sys.partition_functions pf
join sys.partition_range_values prv on
       prv.function_id = pf.function_id and prv.parameter_id = 1
join sys.partition_parameters pp on pp.function_id = pf.function_id
left join sys.partition_schemes ps on ps.function_id = pf.function_id
left join sys.destination_data_spaces dds
    on dds.partition_scheme_id = ps.data_space_id and
       dds.destination_id = prv.boundary_id + cast(pf.boundary_value_on_right as int)
left join sys.filegroups fg on fg.data_space_id = dds.data_space_id
union all
select pf.name as [partition_function],
       ps.name as [partition_scheme],
       pf.fanout as [partition_number],
       case when prv.value is null then NULL else '>' end as [relation],
       prv.value as [boundary_value],
       type_name(pp.system_type_id) as [type],
       fg.name as [filegroup],
       case when ps.name is null then NULL else N'IN USE' end as [status]
from sys.partition_functions pf
join sys.partition_parameters pp on pp.function_id = pf.function_id
left join sys.partition_schemes ps on ps.function_id = pf.function_id
left join sys.destination_data_spaces dds
    on dds.partition_scheme_id = ps.data_space_id and dds.destination_id = pf.fanout
left join sys.filegroups fg on fg.data_space_id = dds.data_space_id
left join sys.partition_range_values prv
    on prv.function_id = pf.function_id and prv.parameter_id = 1 and
       prv.boundary_id = pf.fanout - 1
where pf.boundary_value_on_right = 0
union all
select pf.name as [partition_function],
       ps.name as [partition_scheme],
       NULL, NULL, NULL, NULL,
       fg.name as [filegroup],
       case when dds.destination_id = pf.fanout + 1
            then N'NEXT USED' else N'NOT USED'
       end as [status]
from sys.partition_functions pf
join sys.partition_schemes ps on ps.function_id = pf.function_id
join sys.destination_data_spaces dds
    on dds.partition_scheme_id = ps.data_space_id and
       dds.destination_id > pf.fanout
join sys.filegroups fg on fg.data_space_id = dds.data_space_id

To watch this view working, simply execute:

SELECT * FROM PartitionRanges WHERE partition_function = N'PFTest'

partition_function partition_scheme partition_number relation boundary_value type filegroup status
------------------ ---------------- ---------------- -------- -------------- ---- --------- ---------
PFTest             PSTest           1                <        10             int FG1       IN USE
PFTest             PSTest           2                >=       10             int FG2       IN USE
PFTest             PSTest           3                >=       20             int FG3       IN USE
PFTest             PSTest           4                >=       30             int FG4       IN USE
PFTest             PSTest           5                >=       40             int FG5       IN USE

As one can see, this view returns a row for every partition defined by the partition function together with the range of values covered by this partition and the filegroup the partition is mapped to by the partition scheme. If there is any filegroup assigned in the partition scheme currently not mapped to any partition, it will also be shown together with its status, e.g. NEXT USED.

Performing Alterations on Partitions

Next we will perform modifications on the partition function and study the changes to the metadata. The two possible modifications of a partition function are the deletion of a partition by merging two adjacent partition ranges into one, and the creation of a new partition by splitting one existent partition into two. This is being accomplished by the two variants of the ALTER PARTITION FUNCTION statement.

The first one we are investigating in is the MERGE. The syntax is:

ALTER PARTITION FUNCTION <partition_function_name> () MERGE RANGE (<boundary_value>);

The boundary_value has to be one of those already stored in sys.partition_range_values for this partition function.

If we perform the following MERGE operation on our sample partition function:

ALTER PARTITION FUNCTION PFTest() MERGE RANGE (20)

and then check again the results of our PartionRanges view, we’re getting:

partition_function partition_scheme partition_number relation boundary_value type filegroup status
------------------ ---------------- ---------------- -------- -------------- ---- --------- ---------
PFTest             PSTest           1                <        10             int FG1       IN USE
PFTest             PSTest           2                >=       10             int FG2       IN USE
PFTest             PSTest           3                >=       30             int FG4       IN USE
PFTest             PSTest           4                >=       40             int FG5       IN USE

As one can see, the former partition 3, which had been mapped to filegroup FG3, has been removed and partition 2 now covers the range between 10 and 30, i.e. the dropped partition has been merged with its predecessor partition. This is true because our partition function was defined as a RANGE RIGHT function. For a RANGE LEFT function the partition would have been merged with its successor partition. Since we have not yet partitioned any table using this partition scheme, no real data have been moved by the merge. But otherwise, all data that would have been placed in partition 3 prior to the merge, would have to be moved from FG3 to FG2 during the merge. Obviously, that is something that should be avoided for big partitions. Therefore, usually we only merge partitions that are already empty. Later we will see how a partition easily can be emptied by switching partitions out.

But, how do we drop partition 1? Because, if we specify for the boundary value in the MERGE statement the lowest possible value, which is 10, partition 2 will be dropped, and not partition 1. The answer is, you can’t drop partition 1 if the partition function defines a RANGE RIGHT partition. Therefore, when having the need to regularly switch out the partition with the oldest data, as in a sliding window scenario, and the partition function is defined as RANGE RIGHT, partition 1 should be an empty partition, not containing any data. Hence, when defining the partition scheme we can specify any filegroup, like PRIMARY, for the empty first partition as it never should receive any data.

Because of this, you might argue that RANGE RIGHT partitions are not the best choice for sliding window scenarios. But stay tuned. We will discover a similar problem for LEFT RANGE partitions, just on the opposite end.

Now let’s inspect the effects of the other operation on partition functions, the SPLIT. The syntax is:

ALTER PARTITION FUNCTION <partition_function_name> () SPLIT RANGE (<boundary_value>);

The boundary_value must be a new one, i.e. not yet existing in sys.partition_range_values for this partition function, and the data type of the new boundary value must be implicitly convertible to the type of the partition function parameter.

As the name implies, a SPLIT operation splits an existing partition range into two ranges. All the existing data that fall into the new range have to physically move into the new partition. Again, this should be avoided for big partitions by enforcing that none of the existing data rows falls into the new range, e.g. by means of a CHECK CONSTRAINT. Also note, that even if no data are being moved, the filegroup mapped to the partition to be split must not be designated as READ_ONLY.

Another thing we have to make sure before performing the SPLIT, is that a filegroup marked as NEXT USED exists for all partition schemes based on the partition function that we’d like to split. The new partition being created by the SPLIT implicitly will be mapped to this NEXT USED filegroup. If such a filegroup does not yet exist for a partition scheme, you can create one using the following T-SQL statement:

ALTER PARTITION SCHEME <partition_scheme_name> NEXT USED <filegroup_name>;

You can also designate a NEXT USED filegroup in the CREATE PARTITION SCHEME statement by listing more filegroups than partitions exist. In this case, the first filegroup not mapped to a partition will be marked as NEXT USED.

In our previous MERGE example we have removed filegroup FG3 from the partition scheme. So let’s mark this one as our NEXT USED filegroup:

ALTER PARTITION SCHEME PSTest NEXT USED FG3

Again, let’s see the effect of this statement by querying our PartitionRanges view. Here’s the result:

partition_function partition_scheme partition_number relation boundary_value type filegroup status
------------------ ---------------- ---------------- -------- -------------- ---- --------- ---------
PFTest             PSTest           1                <        10             int FG1       IN USE
PFTest             PSTest           2                >=       10             int FG2       IN USE
PFTest             PSTest           3                >=       30             int FG4       IN USE
PFTest             PSTest           4                >=       40             int FG5       IN USE
PFTest             PSTest           NULL             NULL     NULL           NULL FG3       NEXT USED

As one can see, the partition scheme now has filegroup FG3 designated as the NEXT USED filegroup.

Now we are able to issue a SPLIT operation on our partition function. Here’s the statement we’d like to execute:

ALTER PARTITION FUNCTION PFTest() SPLIT RANGE (50)

This statement splits the existing partition range for values >= 40 into the range >= 40 and < 50 and the range >= 50 and thus adds an additional partition at the right end of our partition ranges. Again, let’s verify this using the PartitionRanges view:

partition_function partition_scheme partition_number relation boundary_value type filegroup status
------------------ ---------------- ---------------- -------- -------------- ---- --------- ---------
PFTest             PSTest           1                <        10             int FG1       IN USE
PFTest             PSTest           2                >=       10             int FG2       IN USE
PFTest             PSTest           3                >=       30             int FG4       IN USE
PFTest             PSTest           4                >=       40             int FG5       IN USE
PFTest             PSTest           5                >=       50             int FG3       IN USE

Previously, I mentioned that we also might run into troubles removing the outmost partition with a LEFT RANGE partition function. So let’s create another RANGE LEFT partition function together with a partition scheme and see what might be the effect of adding a partition at the right end. Here are the statements to create the function and scheme:

CREATE PARTITION FUNCTION PFTestL (int) AS RANGE LEFT FOR VALUES (10, 20, 30)
GO
CREATE PARTITION SCHEME PSTestL AS PARTITION PFTestL TO (FG1, FG2, FG3, FG4, FG5)
GO
SELECT * FROM PartitionRanges WHERE function_name = N'PFTestL'

partition_function partition_scheme partition_number relation boundary_value type filegroup status
------------------ ---------------- ---------------- -------- -------------- ---- --------- ---------
PFTestL            PSTestL          1                <=       10             int FG1       IN USE
PFTestL            PSTestL          2                <=       20             int FG2       IN USE
PFTestL            PSTestL          3                <=       30             int FG3       IN USE
PFTestL            PSTestL          4                >        30             int FG4       IN USE
PFTestL            PSTestL          NULL             NULL     NULL           NULL FG5       NEXT USED

Note, that we already have created a NEXT USED filegroup in the CREATE PARTITION SCHEME statement by specifying more filegroups than required by the underlying partition function. Now let’s see the effect of splitting the rightmost partition by running the following statement and looking at our PartionRanges view output:

ALTER PARTITION FUNCTION PFTestL() SPLIT RANGE (40)

partition_function partition_scheme partition_number relation boundary_value type filegroup status
------------------ ---------------- ---------------- -------- -------------- ---- --------- ---------
PFTestL            PSTestL          1                <=       10             int FG1       IN USE
PFTestL            PSTestL          2                <=       20             int FG2       IN USE
PFTestL            PSTestL          3                <=       30             int FG3       IN USE
PFTestL            PSTestL          4                <=       40             int FG5       IN USE
PFTestL            PSTestL          5                >        40             int FG4       IN USE

As you can see, the new partition has been inserted logically between partition 3 and the previous partition 4, and all the rows from previous partition 4 with values > 30 and <= 40 would have been moved into the new partition 4, i.e. from filegroup FG4 to filegroup FG5. Thus, with a LEFT RANGE partition function you can’t add a new partition at the right end. The answer to this dilemma again is the use of an empty partition at the high end of the partition range when using a LEFT RANGE partition function.

Therefore, the general advice, when defining the partition scheme for a sliding window scenario, is to keep the first partition, for RIGHT RANGE partition functions, or respectively the last partition, for LEFT RANGE partition functions, empty and to map this partition in the partition scheme to a special filegroup, like PRIMARY, in order to make your intention, to keep this partition empty, more clear. Additionally, a CHECK CONSTRAINT can be defined to enforce this requirement.

Hence, in our previous examples the partition schemes should have been defined the following way, if they are intended to be used in a sliding window scenario:

CREATE PARTITION SCHEME PSTest AS PARTITION PFTest TO ([PRIMARY], FG1, FG2, FG3, FG4)

CREATE PARTITION SCHEME PSTestL AS PARTITION PFTestL TO (FG1, FG2, FG3, [PRIMARY], FG4)

Note, that if we specify a NEXT USED filegroup in the partition scheme creation statement for a LEFT RANGE partition function, this filegroup mapping has to appear after the mapping for the empty partition. Therefore, in the second example, the NEXT USED filegroup FG4 has to come after PRIMARY.

Partitions and Allocation Units

So far we have covered all the basics about partitioning. We have seen how to define the partition ranges using a partition function and how to map a filegroup to each partition using a partition scheme. Next, I will talk about the effects on the metadata when partitioning a table or an index.

First, let’s create a table and partition this table based on the partition scheme PSTest defined earlier. We want to make the table definition fairly simple. The table just consists of a unique key column and the column we want the table to be partitioned on. Here’s the table definition:

CREATE TABLE TabTest (pk uniqueidentifier not null, pc int not null) ON PSTest(pc)

Now, let’s explore the changes introduced into the metadata by creating this partitioned table. Every table/index has a data space assigned. As we have seen, a data space either can be a filegroup, or a partition scheme. The sys.indexes row for the table has a column data_space_id which you can use to join sys.indexes with the system catalog view sys.data_spaces. Here you can query the type column to see if the data space refers to a filegroup, if the value is ‘FG’, or to a partition scheme, if the value is ‘PS’. But if you know in advance that you are dealing with a partition scheme, you’d better join with sys.partition_schemes, because this derived view returns only the subset of sys.data_spaces where type is ‘PS’, augmented with the ID of the partition function the scheme is based on.

In order to determine the partitioning column, one has to look in the sys.index_columns view for rows pertaining to this sys.indexes entry and having a non-zero partition_ordinal value.

Given that, the following query returns all partitioned tables and indexes together with the partition scheme and the partitioning column:

select object_name(i.object_id) as [table_name],
       i.index_id, ps.name as [partition_scheme],
       col_name(i.object_id, ic.column_id) as [partitioning_column]
from sys.indexes i
join sys.partition_schemes ps
    on ps.data_space_id = i.data_space_id
join sys.index_columns ic
    on ic.object_id = i.object_id and ic.index_id = i.index_id and
       ic.partition_ordinal > 0

table_name index_id partition_scheme partitioning_column
---------- ----------- ---------------- -------------------
TabTest 0 PSTest pc

Next, we’d like to enter data into the partitioned table. But before doing so, we have to create database files within each of the filegroups defined for the partition scheme. Just add a secondary data file to each of the 5 filegroups we have created previously. For our purpose it’s not important how big those files are and which drive they are placed on. But in practice it deserves some careful planning to specify a file size big enough to avoid auto-grow and small enough to avoid wasting precious disk space. Therefore you need some good estimates on the expected maximum partition size. Also, for optimal performance, it would be beneficial to place each of the files on a separate physical disk.

After having created the disk files, we should enter a few rows in our sample table such that each partition has at least 1 row. But leave the first partition empty, as later we want to use this table for demonstrating a sliding window process (as we have seen in the previous section, for a RANGE RIGHT partition function, the first partition should be left empty).

After having filled the partitions with data, we’d like to see how the data are being distributed over the partitions. Again, there are several system catalog views available to make this information visible. First there is the view sys.partitions, providing an entry for every partition defined in the database (note, that even non-partitioned tables consist of one partition). This entry contains the partition_number, the number of rows within this partition and the hobt_id column, which you can join with the container_id column of the sys.allocation_units view. Each partition can consist of up to three allocation units, for in-row data, LOB data and row-overflow data. From the sys.allocation_units view you can gain information about the number of pages allocated for each allocation unit. If you also want to retrieve information about the actual pages allocated, you’d better join with the undocumented sys.system_internals_allocation_units view instead of sys.allocation_units, because this view additionally returns the first_page, root_page and first_iam_page values for traversing the page chain or index tree.

In order to get this information for our sample table, you can issue the following select statement and get these results returned:

select object_name(p.object_id) as [table_name], p.index_id,
       p.partition_number, p.rows, au.total_pages as [pages],
       au.type_desc as [allocation_type], au.first_page, au.root_page
from sys.partitions p join sys.system_internals_allocation_units au
    on au.container_id = p.hobt_id
where p.object_id = object_id(N'TabTest')

table_name index_id partition_number rows   pages allocation_type first_page     root_page
---------- -------- ---------------- ------ ------ --------------- -------------- --------------
TabTest    0        1                0      0      IN_ROW_DATA     0x000000000000 0x000000000000
TabTest    0        2                10     2      IN_ROW_DATA     0x080000000400 0x000000000000
TabTest    0        3                10     2      IN_ROW_DATA     0x080000000500 0x000000000000
TabTest    0        4                10     2      IN_ROW_DATA     0x080000000600 0x000000000000
TabTest    0        5                10     2      IN_ROW_DATA     0x080000000700 0x000000000000

As you can see, our table has 5 partitions. Partition 1 has no data allocated and all other partitions are having 1 allocation unit assigned for in-row data pages, consisting of 2 pages (1 data and 1 IAM page). Since our table still is a heap, the root page pointer is NULL.

Partitioned Indexes

In the next section we will have a closer look at partitioned indexes. Besides partitioning tables, you may also partition indexes. Partitioning an index transforms the single B-tree of a non-partitioned index into several smaller B-trees, one for each partition defined by the partition function, where each B-tree only consists of index rows where the partitioning column falls into the partition’s range. Therefore, the partitioning column always has to be part of a partitioned index.

Splitting a partitioned index into several independent B-trees obviously also has consequences for index seeks. As long as the partitioning column is one of the search predicates, the optimizer knows which of the trees to choose for the index seek. But if the filter criteria do not include the partitioning column, the optimizer would not know where to start the index seek or scan when using a partitioned index. E.g. if we create the following composite partitioned clustered index “CREATE CLUSTERED INDEX IX_Tab ON Tab (a, b)”, where table Tab is being partitioned on column b, the query “SELECT * FROM Tab WHERE a = @x” cannot be performed as a simple clustered index seek, like in the non-partitioned case, but each of the index trees would have to be traversed separately to find the qualifying records. As this example shows, for partitioned tables the rule, that the table rows are sorted by the clustered index keys, does not necessarily hold true for the whole table, but only for single partitions. Therefore, it’s always good practice to include the partitioning column in your search predicates, even if it does not affect the result set.

If an index is partitioned on the same column as its base table and the partition scheme used is based on the same partition function used for partitioning the table, both table and index are said to be aligned. If you do not specify otherwise, an index is automatically aligned with its table. A clustered index always has to be aligned with its base table. In fact, if you create the clustered index differently, the underlying table will adopt the allocation scheme of the clustered index. Thus, to enforce a new allocation scheme for a table, just rebuild the clustered index with the new scheme. Therefore, the easiest way to partition an existing table is to create a partitioned clustered index on the table, although, it might not be the most efficient way doing so.

Having all indexes aligned with its base table is essential for switching table partitions in and out, as we’ll see later. But it’s also legal to partition an index on a different partitioning column or using a different partition function with a different number of partitions or different boundary values. You may also create an index, that is not partitioned, on a partitioned table, or partition an index although the base table isn’t partitioned. If you want to apply a different allocation scheme on the index, just use the ON clause in the index creation statement just like for the table creation statement.

As we have seen, a partitioned index always has to include it’s partitioning column. There are a few rules to be aware of concerning the inclusion of the partitioning column:

For every unique index (also implies primary key and unique constraints), the partitioning column must be part of its index keys.
For a non-unique clustered index, that does not contain the partitioning column, that column implicitly is being added to the index keys.
For a non-unique nonclustered index, that does not include the partitioning column, the partitioning column is added at the leaf level as a non-key column.

The major consequence is, that you cannot partition an unique index on a column that is not one of the index keys. Therefore, if you need an unique index or constraint to enforce the uniqueness of a set of columns, that does not include the partitioning column, it will be necessary to make the unique index a non-partitioned one, or find other ways to check for uniqueness.

Now let’s create two indexes on our sample table, a clustered one on the partitioning column and a primary key constraint on column pk, which is not partitioned:

CREATE CLUSTERED INDEX CIX_TabTest ON TabTest (pc)
GO
ALTER TABLE TabTest ADD CONSTRAINT PK_TabTest PRIMARY KEY (pk) ON FG1

And with query from the previous chapter we can inspect again the allocations:

table_name index_id partition_number rows   pages allocation_type first_page     root_page
---------- -------- ---------------- ------ ------ --------------- -------------- --------------
TabTest    1        1                0      0      IN_ROW_DATA     0x000000000000 0x000000000000
TabTest    1        2                10     2      IN_ROW_DATA     0x0B0000000400 0x0B0000000400
TabTest    1        3                10     2      IN_ROW_DATA     0x0B0000000500 0x0B0000000500
TabTest    1        4                10     2      IN_ROW_DATA     0x0B0000000600 0x0B0000000600
TabTest    1        5                10     2      IN_ROW_DATA     0x0B0000000700 0x0B0000000700
TabTest    2        1               40     2      IN_ROW_DATA     0x100000000300 0x100000000300

As you can see the clustered index is partitioned the same way as its base table and is spread over the partitions 2 through 5, with a separate index root page on every partition. Thus, the one clustered index is divided into 4 separate B-trees. Whereas the index for the primary key (index_id = 2) is not partitioned. But at the leaf level this index still has the partitioning column included as this is the clustering key, and therefore, when performing the bookmark lookup to get to the data page, SQL Server immediately knows in which partition to look for.

Sliding Window Scenario

One of the major benefits of partitioning is the ability to efficiently move chunks of data in and out of a table. If the chunks happen to be multiples of a partition, no data physically have to be moved, and only metadata for allocations have to be adjusted. This fact is being exploited in sliding window scenarios where on a regular basis (e.g. quarterly, monthly, weekly, or even daily) aged data are being moved out of a table, e.g. for archiving, and new data swapped in, that up to this point have been accumulated in a staging table. By collecting the data in a staging table instead of inserting directly into the target table, one can take advantage of the fact, that bulk copying data into an empty staging table with no indexes in place is much more efficient than inserting data directly into a big table with all indexes in place. (See “SQL Server 2000 Incremental Bulk Load Case Study” in https://technet.microsoft.com/library/cc917716.aspx, which is equally applicable to SQL Server 2005 and later).

A typical requirement for a sliding window scenario might look like this:

The last 12 month of business transaction data should be stored in a partitioned fact table of a relational data warehouse.
During each month, the current transaction data from various sources, e.g. the OLTP databases at each business site, are being collected within a staging table of the data warehouse.
On the first of each month, all data older than 12 month should be archived and the data of the previous month should be transferred from the staging table into the fact table.

To put these requirements into reality, we first have to build the partitioned fact table. Either we build that table up from scratch by adding one partition each month until we have reached the final size of 12 month worth of data, or we have the data already available somewhere else, let’s say in a non-partitioned table, and simply have to transition this table into a partitioned table (e.g. by creating a new partitioned clustered index on the table). Either way, we first have to decide on how to distribute the 12 partitions over our storage media. Ideally, we devote one filegroup for every partition. Next, the 12 filegroups have to be spread over the available disk drives. Let’s say 4 drives are available, so we assign each of the filegroups in a round robin fashion to one of the drives by creating a secondary data file for the filegroup on this drive. The files should be sized big enough to hold one month of data plus some room for future growth. Creating the partition function is straight forward. Just define the 12 boundary values, being either the first day or the last day of each month, depending on whether we plan to create a RIGHT or LEFT range function.

Now having the previous 12 month data within the partitioned table, let’s start the preparation work for the next month. We need another filegroup with a data file assigned where we create the non-partitioned staging table. Filling this staging table can be a continuous process during the month, inserting each days transactions at the end of the day, or the staging table is being filled with data at the end of the month (for alternative load scenarios also see the SQL Server Best Practices article “Loading Bulk Data into a Partitioned Table” in https://technet.microsoft.com/library/cc966380.aspx). Nevertheless, the staging table needs to have the same structure as the partitioned table. Whether you add the same indexes to the staging table already at the beginning or after filling the table with data, depends on the usage of the staging table during the month. If you just insert new data, for optimal bulk insert performance, you should refrain from creating the indexes before all data have been loaded into the table. But, if the staging table already participates in queries, some indexes might be essential. Anyway, before merging the staging table with the partitioned table, the staging table must have the same indexes and foreign key constraints as the partitioned table and a check constraint must be defined on the partitioning column, that restricts the data in the staging table to values conformant with the boundaries of new partition that will absorb the data.

Now we have reached the end of the month and during the coming night the staging table’s data need to be moved into the partitioned table, so that this month’s data is available at the beginning of next business day for starting various data analysis jobs. We start with preparing the staging table. We create all the necessary indexes and foreign key constraints and define a check constraint for the partitioning column. Afterwards, we might try to shrink the file(s) associated with the staging table’s filegroup and run a final defragmentation on the staging table. Finally, it also might be a good idea checking the consistency of the table by running a DBCC CHECKTABLE. Now the time has come to move the data.

The essential operation to perform this data movement is the partition switch operation. In T-SQL this is being accomplished by the ALTER TABLE…SWITCH statement. The syntax is:

ALTER TABLE source_table_name SWITCH [PARTITION partition_out] TO target_table_name [PARTITION partition_in]

This statement supports three different scenarios:

You can move a partition of a partitioned table out into a non-partitioned table
You can move a non-partitioned table into a partition of a partitioned table
You can move a partition of one partitioned table into a partition of another partitioned table

All participating tables need to be structurally equivalent, i.e. need to have the same columns, same clustered index, etc. The target table/partition must be empty, and source and target partition must reside in the same filegroup. If the switch target is a partitioned table, all indexes on this table have to be aligned with the table. Any non-aligned index has to be dropped or disabled prior to the switch operation. For an exact list of requirements see the topic ‘Transferring Data Efficiently by Using Partition Switching’ in SQL Server Books Online.

Here’s the list of steps you would have to perform moving the data to/from the partitioned table:

Determine the filegroup where the partition with the oldest data resides (this is partition 1 for a LEFT RANGE partition or partition 2 for a RIGHT RANGE partition).
If this filegroup is marked as READ_ONLY, change it to READ_WRITE.
On this filegroup create a non-partitioned archive table, which is an exact copy of the partitioned table.
If the partitioned table is clustered, create the same clustered index on the archive table.
Using the ALTER TABLE…SWITCH command, switch the partition identified in step 1 into the archive table created in step 3.
Using the ALTER PARTITION FUNCTION command, merge the partition with the lowest partition boundary value, thus dropping the partition switched out in step 5.
If not done yet, for the partition scheme mark the filegroup, where the staging table resides, as NEXT USED using the ALTER PARTITION SCHEME command. (Note, that all partition schemes based on the same partition function need to have a NEXT USED filegroup.)
If the filegroup assigned to the last partition of a RIGHT range partition is marked as READ_ONLY, change it to READ_WRITE.
Add a new partition using the ALTER PARTITION FUNCTION…SPLIT command with the new boundary value.
If CHECK constraints exist on the partitioned table that restrict the possible values of the partitioning column, adjust them accordingly (actually, only the upper bounds need to be adjusted).
Determine the partition number of the new partition added in step 9 (this is the last partition for a RIGHT RANGE partition, or its predecessor for a LEFT RANGE partition).
Switch the staging table into the partition identified in step 11 using the ALTER TABLE…SWITCH command.
Now you can drop the staging table and you might seal the filegroup by designating it as READ_ONLY.
Create a new staging table for the upcoming month, possibly in the filegroup identified in step 1.
Update statistics on the partitioned table.
Save and drop the archive table created in step 3 (this step does not have to be performed immediately and can be deferred to a later point in time).

Although this might sound complex, most of it can be scripted. And best of it, these steps (except the last two) can be executed within seconds as no data really are moved and only metadata have to be changed. See https://sqlpartitionmgmt.codeplex.com for a program that might be helpful for performing these steps.

Enhancements in SQL Server 2008 with Regard to Table and Index Partitioning

SQL Server 2008 provides several enhancements that affect partitioned tables and indexes. Here I’ll discuss some of those enhanced features. See also https://technet.microsoft.com/magazine/2008.04.dwperformance.aspx.

Enhanced parallel query execution

In SQL Server 2005, if a query covers more than one partition, only a single thread is being used for executing the query on each partition in parallel. This limits the parallelism of a query in case the number of partitions touched by the query is less than the number of processors available to SQL Server. SQL Server 2008 removes this restriction and distributes the available threads for query processing evenly on all partitions participating in the query.

For a detailed discussion of this topic see “Parallel Query Execution Strategy for Partitioned Objects” under “Query Processing Enhancements on Partitioned Tables and Indexes” in SQL Server 2008 Books Online.

Partition-aligned indexed views

In SQL Server 2005 you are able to create indexed views on a partitioned table. But these indexed views cannot be partitioned. Hence, before switching partitions you would have to drop the indexed views before the switch and recreate them afterwards. This obviously leaves indexed views on partitioned tables useless in case partition switching is a requirement.

Now, SQL Server 2008 allows to partition indexed views as well. Principally, you define the indexed view the same way you do it in SQL Server 2005. There are a few restrictions concerning the inclusion of the partitioning column in the view definition (see “Partition Switching When Indexed Views Are Defined” in SQL Server 2008 Books Online). Then, when materializing the view by creating the unique clustered index on the view, simply use the ON clause to specify the partition scheme and partitioning column. The same rules for partition alignment apply as for partitioned indexes.

Data Compression

SQL Server 2008 allows for compressing data on a row or page level. The good news is, that this feature not only can be set on a whole table or index, but for each partition individually. Thereby, you can enable the more costly page level compression on older partitions, that are not altered very often, and leave the actual partition, that gets all the new data, uncompressed or row compressed. You can change the compression state of a partition using the ALTER TABLE .. REBUILD PARTITION statement.

For tables with mixed compression settings for different partitions, you should be aware of https://support.microsoft.com/kb/957810.

Lock Escalation

Instead of escalating the existing fine-grain locks on a table to a single table lock in case the lock escalation threshold has been exceeded, SQL Server 2008 allows to escalate the locks on a single partition into a lock on the heap or B-tree level, thus allowing for greater concurrency than a full table lock.

This feature can be set for each table individually by specifying SET (LOCK_ESCALATION = AUTO) with the ALTER TABLE statement.

Support for partitioning in SQL Server Management Studio

The support for partitioning, which Management Studio in SQL Server 2005 provides, is pretty sparse. For instance, you can list and delete the partition schemes and functions defined for a database. But that’s it. Management Studio in SQL Server 2008 extends this functionality by providing several Wizards that may help in creating and maintaining partitioned tables. Just right-click a table in the Object Explorer and select Storage in the context menu. Depending on whether the table is already partitioned or not, you can activate the Create Partition Wizard or the Manage Partition Wizard. These wizards allow you to graphically design partition functions and schemes, create a staging table, or to manage partition switching in a sliding window scenario. Although the wizards are quite neat, you always should be alert if the solution to a complex problem is just a few mouse clicks away. The solutions chosen by the wizards might not always be the most practical ones, e.g. if you ask the wizard to partition a table, the solution proposed by the wizard, although correct, certainly is not the one you’d apply to a existing huge table coming with a bunch of indexes. You’d better script the solution first for further examination, instead of immediately running it. But the wizards come in handy during the design phase of a database or for becoming acquainted with partitioning.

Partitioning of tables with FILESTREAM data

SQL Server 2008 incorporates a new storage scheme for large binary data. If you define a column of type VARBINARY(MAX) with the FILESTREAM storage attribute, the column data will be stored in the servers file system instead of database storage. The locations within the file system where these data are placed, are defined by special FILESTREAM filegroups. Within the CREATE TABLE statement you have to specify in a FILESTREAM_ON clause the filegroup to be used for the FILESTRAEM data in the table. But, if the table is partitioned, also the FILESTREAM storage is going to be partitioned. Therefore, you have to define a separate partition scheme, based on the same partition function and same partitioning column as the table, and mapping each partition to a FILESTREAM filegroup. And this partition scheme has to be used in the FILESTREAM_ON clause instead.

But there is one more requirement for FILESTREAM columns that lessens the usefulness of partitioning tables with FILESTREAM data. A table with FILESTREAM data also needs to have a column with the ROWGUIDCOL property and an UNIQUE or PRIMARY KEY single-column constraint defined on this column. But as you recall, unique partitioned indexes also have to add the partitioning column to their index keys. Hence, the index for this UNIQUE constraint cannot be partitioned. But as you have learned, an index automatically inherits the storage scheme of its base table and this also applies to UNIQUE constraints defined with the table. Thus, in order to make an UNIQUE constraint within a partitioned table a non-partitioned one, you explicitly have to specify a filegroup to store this index on within the CONSTRAINT definition.

But the more important catch here is, that you cannot switch partitions in or out while there is a non-aligned index defined on the table. And you cannot simply drop the UNIQUE constraint or disable the pertaining index while performing the switch, as long as there are FILESTREAM data in the table.