Let us assume we have a mirroring setup, as described in the below image. SQLServer1 hosts the Principal Database (MyDB) while the same database on SQLServer2 is the Mirror Database.

It is possible to create a Snapshot of the Mirrored Database (MyDB on SQLServer2), in case we want to offload some of the Reporting Load from the Principal Database. Of course, we have to remember that the Snapshot presents data specific to a point in time; hence, users will be reading stale data till the Snapshot is refreshed (dropped and re-created).

Pre-requisites for being able to create a Snapshot of the Mirrored Database are:

• The Principal and Mirror should be in Connected State.
• The Principal and Mirror should be completely Synchronized at the time when the Snapshot is being created.

The Mirroring Mode (Asynchronous, Synchronous, and Synchronous with Automatic Failover) are not important as long as the pre-requisites above are true. However, in high transaction rate OLTP systems, with Asynchronous Mirroring setup, it is difficult to find a point in time when the Principal and Mirror are in a Synchronized state; hence, it is very difficult to create Snapshot on the Mirrored Database. If the databases are not synchronized, and we attempt to create a Snapshot on the Mirror Database, we will receive an error as below:

Msg 1822, Level 16, State 1, Line 1
The database must be online to have a database snapshot.

The error can be misleading; because it says nothing about the Mirroring State issue that causes the Snapshot creation to fail; however, we have to keep in mind that error messages related to Database Mirroring are not very descriptive at the moment. The SQL Server Development Team is working on to make the error messages more meaningful, and it may take a while before we see more descriptive messages.

When it comes to the internals of how Database Snapshots work and how the Buffer Pool reacts to queries on the Snapshot Database, the same principals as mentioned in my previous posts still hold true. Only differences being:

• the Reads happen, not from the Online Principal Database, but from the Mirror Database, and
• the Copy-On-Write mechanism transfers pages from the Restoring Mirror Database and not the Online Principal Database.

If you are wondering how the Reads and Writes can happen from a Database that is in Restoring State, you must realize that the State of the Database is how the SQL Server Engine exposes the Logical Database for external users. The Physical Database Files, on the disk, are still available for Reads and Writes, and the SQL Server Engine does Read data from and Write data into these file. Additionally, there are quite a few undocumented DBCC commands that can work against the Restoring Database.

What happens when a role switching occurs is more important. During role switching, the Snapshot is not moved to the new Mirror Server. If the Snapshot was created on MyDB Database on SQLServer2 (in the image above), the Snapshot will still continue to exist on SQLServer2 even after SQLServer2 becomes the Principal Server. Now, if another role switch happens and SQLServer2 becomes the Mirror Server again, the Snapshot will still continue to be on SQLServer2. However, each Role Switch will cause the Database and the Snapshot to be restarted, and hence users will be temporarily disconnected.

One interesting question that was asked to me by one of my customers was: How can I give access to the Snapshot to a particular user without giving him access to the Principal Database?

The answer is not so straight-forward. The Snapshot is a Read-Only copy of the Database; and you cannot simply create an user in the Snapshot. Below are the steps that can be followed for granting access:

For Windows Logins:

• Use sp_grantlogin to grant the Windows Account access to the Principal Server.
• Use sp_grantlogin to grant the Windows Account access to the Mirror Server.
• Add the login as an user in the Principal Database with appropriate privileges. This change will be automatically propagated to the Mirror Database.
• Now, create a Snapshot of the Mirror Database.
• Now, drop the User and the Login from the Principal Database / Server.

For SQL Logins:

• Use sp_addlogin to add the login to the Principal Server.
• Add the login as an user to the Principal Database.
• Use KB918992 to move this login (along with the Password and the SID) to the Mirror Server.
• Now, create a Snapshot of the Mirror Database.
• Drop the User and Login from the Principal Database / Server.

Remember, this is not an One-Time activity. The steps outlined above will have to be repeated each time you refresh (drop and recreate) the Snapshot.

Disclaimer: All information provided here is my personal opinion and is neither verified nor approved by Microsoft before it is published. All information, and code samples, if any, is provided "AS IS" with no warranties and confers no rights.

Before we start drilling down further, let me take a while to explain how we will pull up details about pages in the Buffer Pool. We have a DMV and a few Catalog Views that we can use:

• sys.dm_os_buffer_descriptors: This DMV returns information about all the data pages that are currently in the SQL Server buffer pool. When a data page is read from disk, the page is copied into the SQL Server buffer pool and cached for reuse. Each cached data page has one buffer descriptor. Buffer descriptors uniquely identify each data page that is currently cached in an instance of SQL Server. sys.dm_os_buffer_descriptors returns cached pages for all user and system databases. This includes pages that are associated with the Resource database.
• sys.allocation_units: Contains a row for each allocation unit in the database. sys.dm_os_buffer_descriptors can be joined with sys.allocation_units based on the allocation_unit_id column values.
• sys.partitions: Contains a row for each partition of all the tables and most types of indexes in the database. Special index types like Fulltext, Spatial, and XML are not included in this view. All tables and indexes in SQL Server 2008 contain at least one partition, whether or not they are explicitly partitioned. The container_id column of sys.allocation_units can be joined with the hobt_id column of sys.partitions when the allocation type in sys.allocation_units is either 1 (In-row data) or 3 (Row-overflow data). When allocation type in sys.allocation_units is 3 (Large object (LOB) data (text, ntext, image, xml, large value types, and CLR user-defined types)), container_id column of sys.allocation_units can be joined with the partition_id column of sys.partitions.

Joining these, we can construct the following query that can give us the Count of Pages in the Buffer Pool on a Per Object and Page Type basis:

/*
This code has to be run under the context of the Database as
in Line 2. For example, if you want to get the details of the
database "MyDB", change Line 2 to DB_ID('MyDB'), and execute
this under the context of MyDB.
*/
DECLARE @DBID INT
SELECT @DBID = DB_ID('Database_Name')
SELECT 
    M.[object_name] [Object Name]
    , M.[page_type] [Page Type]
    , COUNT(M.[page_type]) AS [Page_Count]
FROM
(
    SELECT 
        OSBD.[database_id]
        , OSBD.[allocation_unit_id]
        , OSBD.[page_type]
        , AU.[container_id]
        , P.[object_id]
        , S.[name] + '.' + O.[name] AS [object_name]
    FROM sys.dm_os_buffer_descriptors OSBD
    JOIN sys.allocation_units AU
    ON OSBD.allocation_unit_id = AU.allocation_unit_id
    JOIN sys.partitions P
    ON AU.[container_id] = P.[hobt_id]
    JOIN sys.objects O
    ON P.[object_id] = O.[object_id]
    JOIN sys.schemas S
    ON O.[schema_id] = S.[schema_id]
    WHERE OSBD.database_id = @DBID
    AND AU.[type] IN (1, 3)
    UNION ALL
    SELECT 
        OSBD.[database_id]
        , OSBD.[allocation_unit_id]
        , OSBD.[page_type]
        , AU.[container_id]
        , P.[object_id]
        , S.[name] + '.' + O.[name]
    FROM sys.dm_os_buffer_descriptors OSBD
    JOIN sys.allocation_units AU
    ON OSBD.allocation_unit_id = AU.allocation_unit_id
    JOIN sys.partitions P
    ON AU.[container_id] = P.[partition_id]
    JOIN sys.objects O
    ON P.[object_id] = O.[object_id]
    JOIN sys.schemas S
    ON O.[schema_id] = S.[schema_id]
    WHERE OSBD.database_id = @DBID
    AND AU.[type] IN (2)
) AS M
GROUP BY M.[object_name], M.[page_type]
ORDER BY 1

Let us now see what happens when we go ahead and create a Snapshot of a database. Before we create the Snapshot, we will clear off the Buffer Pool using the DBCC DROPCLEANBUFFERS command.

Note: After running the DBCC DROPCLEANBUFFERS command, make sure you execute the query mentioned above to ensure that the Buffer Pool is completely empty of User Objects. Few pages of system tables may continue to exist in the Buffer Pool, and this is fine.

After we create the Database Snapshot, if we execute the query above, on both the Source Database and the Snapshot, we will see that the Buffer Pool contains NO user data or index pages. All pages for these databases will belong to the system tables, which indicates that some of the metadata is transformed to the Snapshot.

Note: In case you have defined Full Text Indexes on your tables, this will not hold good – the query against the Source Database will show some pages belonging to some of the User Tables.

We will now query ONE row from one of the User Tables in the Snapshot. The table I am using is a heap with no indexes defined on it for simplicity, to avoid taking into consideration the Index Pages. You can try querying tables with Indexes (and even use complex queries so that multiple tables and indexes are touched) to observe the behavior.

As we have discussed in my earlier post, the pages in the Source Database have not changed yet, and hence the Snapshot contains no user data. So, now, the data has to be retrieved from the Source Database. The SQL Server Engine checks if the requested pages are already in the Buffer Pool under the context of the Snapshot Database. If yes, the data is returned back from the Buffer Pool without any additional work. However, if the pages are not already cached in the Buffer Pool, an In-Memory bitmap is consulted to find out which of the requested pages are available in the Sparse File, and which have to be read from the Source Database. IO requests are issued accordingly. This In-Memory bitmap resides in the Memory till the Snapshot is dropped. If SQL Server is recycled within the life-span of the Snapshot, the metadata of the Snapshot is queried to generate the bitmap again after SQL Server comes online. Since the Source Database has not been queried, the pages will not be cached under the Source Database; if later a different query needs this data from the Source Database, another set of IO requests will follow.

The same query (with the database name in Line 2 changed), when run against the Snapshot database, shows that the data pages traversed by the query are indeed cached in the Buffer Pool under the context of the Snapshot Database.

Remember: Databases do not share pages in the Buffer Pool.

Now, let us see what happens when we modify a row in the Source Database. We will again use the Table with No Indexes for simplicity, and before we execute the update statement, we will clean up the Buffer Pool using the DBCC DROPCLEANBUFFERS command.

When we execute the query to modify the data on the Source Database, the data pages are first cached into the Buffer Pool under the context of the Source Database, if not already cached. Also, since we have to do a Table Scan for locating the one row that we want to modify, all these traversed pages will be cached. Now, the Copy On Write mechanism kicks in and the to-be-dirtied pages are copied into the Snapshot sparse file. Once the copy is complete, the pages can be modified in the Buffer Pool. The query against the Source Database clearly shows that the pages are cached in the context of the Source Database. When we execute the query against the Snapshot Database, we see that none of the pages are cached in the Buffer Pool under the context of the Snapshot. This is kind of expected now, as we have already established that databases do not share pages in the Buffer Pool, and pages are cached in the Buffer Pool under the context of the database which read those pages from the disk. Note: The above experiments show that if the same pages of the database are read from the Source Database as well as from the Snapshot, two copies of the pages will be cached in the Buffer Pool – one set under the context of the Source Database, and the second set under the context of the Snapshot. This might mean a waste of Buffer Pool memory, especially if these pages have never been modified since the snapshot creation.

We will do another experiment this time. We will modify a page in the Source Database and read the same page from the Snapshot. Later, we will check the contents in the Buffer Pool for this one object for both databases. For this purpose, we will use a slightly different query, as pasted below:

/*
This code has to be run under the context of the Database as
in Line 2. For example, if you want to get the details of the
database "MyDB", change Line 2 to DB_ID('MyDB'), and execute
this under the context of MyDB.
*/
DECLARE @DBID INT
SELECT @DBID = DB_ID('Database_Name')
SELECT 
    BP.[Database Name]
    , BP.[Object Name]
    , BP.[File ID]
    , BP.[Page ID]
    , BP.[Page Type]
    , BP.[Number of Rows]
    , BP.[Is Modified]
FROM
(
    SELECT 
        DB_NAME(OSBD.[database_id]) [Database Name]
        , S.[name] + '.' + O.[name] [Object Name]
        , OSBD.[file_id] [File ID]
        , OSBD.[page_id] [Page ID]
        , OSBD.[page_type] [Page Type]
        , OSBD.[row_count] [Number of Rows]
        , OSBD.[is_modified] [Is Modified]
    FROM sys.dm_os_buffer_descriptors OSBD
    JOIN sys.allocation_units AU
    ON OSBD.allocation_unit_id = AU.allocation_unit_id
    JOIN sys.partitions P
    ON AU.[container_id] = P.[hobt_id]
    JOIN sys.objects O
    ON P.[object_id] = O.[object_id]
    JOIN sys.schemas S
    ON O.[schema_id] = S.[schema_id]
    WHERE OSBD.database_id = @DBID
    AND AU.[type] IN (1, 3)
    UNION ALL
    SELECT 
        DB_NAME(OSBD.[database_id]) [Database Name]
        , S.[name] + '.' + O.[name] [Object Name]
        , OSBD.[file_id] [File ID]
        , OSBD.[page_id] [Page ID]
        , OSBD.[page_type] [Page Type]
        , OSBD.[row_count] [Number of Rows]
        , OSBD.[is_modified] [Is Modified]
    FROM sys.dm_os_buffer_descriptors OSBD
    JOIN sys.allocation_units AU
    ON OSBD.allocation_unit_id = AU.allocation_unit_id
    JOIN sys.partitions P
    ON AU.[container_id] = P.[partition_id]
    JOIN sys.objects O
    ON P.[object_id] = O.[object_id]
    JOIN sys.schemas S
    ON O.[schema_id] = S.[schema_id]
    WHERE OSBD.database_id = @DBID
    AND AU.[type] IN (2)
) AS BP
WHERE BP.[Object Name] = 'SchemaName.TableName'
ORDER BY BP.[File ID], BP.[Page ID]

When we analyse the contents of the Buffer Pool, we will see that the modified page has been marked as Modified in the Source Database, and marked as Clean in the Snapshot (screenshots below). This conclusively proves that pages in the Buffer Pool are not shared and that different states of the same page might be present in the Buffer Pool depending on when the Snapshot was created and when the page was modified.

Note: The page marked as Modified will not be removed from the Buffer Pool until Checkpoint runs on the database and flushes the dirty page to the disk. Once the dirty page is flushed, the page is marked as Clean (the Is_Modified bit is cleared), and then this page can be removed from the Buffer Pool by using the DBCC DROPCLEANBUFFERS command.

Summary: Databases do not share pages (memory) in the Buffer Pool. Each database (Source Database and the Snapshot Database) caches its own pages in the Buffer Pool depending on the data pages being read. Hence, if you are planning to create multiple snapshots of the same database, apart from considering the impact due to increased IO (as described in my earlier post), you will also need to consider the increased Buffer Pool usage, and impact of it on performance.

In the next post, we will see what happens internally when we create Snapshots on Mirror databases.

Disclaimer: All information provided here is my personal opinion and is neither verified nor approved by Microsoft before it is published. All information, and code samples, if any, is provided "AS IS" with no warranties and confers no rights.

All credit for this post goes to my colleague, Mohammad Sufian, without whose help, this post may have not been possible.

So, what are Database Snapshots? According to the Books OnLine, “A database snapshot provides a read-only, static view of a source database as it existed at snapshot creation, minus any uncommitted transactions. Uncommitted transactions are rolled back in a newly created database snapshot because the Database Engine runs recovery after the snapshot has been created (transactions in the database are not affected).” This means that the database snapshot is a transactionally consistent image of the database as of the snapshot creation time.

What can I use Database Snapshots for? Again, according to Books OnLine, some uses of Database Snapshots might be:

• Maintaining historical data for report generation.
• Using snapshots on Mirrored databases can help in offloading reporting.
• Safeguarding data against administrative and user errors.
• Managing a test database.

What can’t I do in Database Snapshots? The following lists a few things that are not allowed or will not work in Database Snapshots:

• Database snapshots are specific to a SQL Server instance.
• Database snapshots can be created only if the source database is ONLINE or if it is the mirrored database.
• Database snapshots cannot be backed up or restored.
• Database snapshots cannot be detached or attached.
• Database snapshots are read-only so they cannot be renamed. Settings cannot be modified as well.
• Database snapshots inherit security settings, filegroup state from the source database and these settings cannot be modified on the database snapshot.
• Database snapshot file specifications cannot be modified.
• Database snapshot is marked suspect if there is no disk space when a modified page is being copied from the source database.
• The point-in-time of a database snapshot only approximate in nature and can be determined from the create_date column of sys.databases catalog view entry for the database snapshot.
• Database snapshots created on two source databases that have say views pointing from one database to another will retain their original definition in the snapshots also. Any queries against those views in the database snapshots will refer to the source databases if they are online. If the source databases are mirrored to a different server and database snapshots are created against those then the queries will fail since the source databases will be in recovery mode.
• Full-text indexing is not supported on database snapshots.
• Database snapshots of system databases are not allowed.

How can we create a Snapshot and what happens when we create a snapshot? To create a snapshot of a database we can use the CREATE DATABASE <specifications> AS SNAPSHOT OF <DatabaseName> command. Below is an example of the command that I used to create a Snapshot of the AdventureWorks database:

CREATE DATABASE AdventureWorks_Snapshot
ON 
(
    NAME = AdventureWorks2008R2_Data,
    FILENAME = 'D:\AdventureWorks_Snapshot.ss'
) AS SNAPSHOT OF AdventureWorks
GO

Remember: To be able to understand all of the below discussion, we need a database that is not touched by user transactions. Hence, I am using the AdventureWorks database on my local machine.

Don’t believe it? Let’s see… The query below will give us the Max Size and the Size on Disk for the Snapshot Database:

DECLARE @DBID INT
SELECT @DBID = DB_ID('AdventureWorks_Snapshot')
SELECT 
    DB_NAME(@DBID) AS [Database Name],
    VFS.[file_id] AS [FileID],
    MF.[name] AS [Logical File Name],
    (MF.[size] * 8) AS [Max Size (KB)],
    (VFS.size_on_disk_bytes / 1024) AS [Size on Disk (KB)]
FROM sys.dm_io_virtual_file_stats(@DBID, -1) VFS
JOIN sys.master_files MF
    ON VFS.[file_id] = MF.[file_id]
WHERE VFS.database_id = MF.database_id
AND MF.database_id = @DBID
AND MF.type_desc != 'LOG'
GO

Till now, the AdventureWorks_Snapshot.ss sparse file that we created contains no user data – just the necessary metadata for the database. The Max Size of the Snapshot is equal to the Size of the database when the snapshot was created, and the Size on Disk is the actual space used by the snapshot. Hence, 3200 KB or just a little above 3 MB is allocated, and this is sufficient for maintaining the metadata of the database. So, now, the obvious question is, how will data be returned if we query the snapshot? This is clearly documented in Books OnLine:

Since we have no user data in the snapshot at present and the data pages in the Source Database has not changed, the Read operation on the Snapshot reads the original page from the source database to retrieve the data from that page. The metadata in the Snapshot has the information about whether a page has been copied from the source; and based on this information, reads are redirected. This is, in fact, a very good design, as otherwise, we would have to copy huge sets of data (GBs or even TBs of data) to the Snapshot file. The copy operation would have taken huge time to complete, and would have exerted a huge load on the disk. Myth 1: The sparse file contains all zero-d out pages. Contrary to common belief, the sparse file does not contain zero-d out pages just yet. No extents are allocated, and the sparse file contains no data pages.

Now, when a page is modified on the source database, the pre-modified copy of the page is first copied to the database snapshot, and then the original page is modified. This operation is irrespective of whether the transaction that modified the page commits or rolls back. Here are the steps that are followed when a page is modified or dirtied:

• The Page is prepared to be dirtied.
• An EX Latch is acquired on the page.
• Replica writes are completed – this means that a replica of the page is created in the snapshot sparse file.
• Changes are now made to the page.
• The EX Latch on the page is now released.

This operation is known as the Copy On Write (COW) operation on database snapshots.

Once a page has been copied to the Snapshot, the metadata in the Snapshot is changed to reflect that the original state of the page is now available in the snapshot. Now, when a query attempts to read this page from the Snapshot (by querying the Snapshot Database), the page is returned directly from the Snapshot. For all other pages that have not been copied to the Snapshot, the source database will still be read to fetch the data. Note: Certain operations on the data, such as a Truncate Table, do not modify the data pages; rather, the allocations related to the table are removed from the metadata. So, when you run Truncate Table on a database, and cause a huge table (say 400 GB) to be emptied, the pages (data and index) belonging to the table are not copied over. When this same table or any other object attempts to re-use these allocations and over-write the pages – that’s when these pages are actually copied to the snapshot.

Myth 2: Modified pages are placed adjacent to each other in the snapshot to reduce the sparse file used size.
Contrary to common belief, modified pages are not placed adjacent to each other. Let us assume, Pages 50 and 150 are being modified. Let us also assume Page 50 belongs to Extent 7 and Page 150 belongs to Extent 19. So, Page 50 is the 2nd Page of Extent 7 and Page 150 is the 6th page of Extent 19.

When Page 50 is dirtied, a complete allocation of Extent 7 will be created in the sparse file at storage offset ((7 – 1) * 8 + 1) * 8192 bytes, and the contents of Page 50 will be copied to the 2nd Page in this extent. All other pages will be zero-d out. So, essentially, offset of the Page 50 will be 50 * 8192 bytes. When Page 51 is dirtied, a new extent will not be allocated; Extent 7 already has the place-holder for the Page 51, and hence, when this page is dirtied on the source, it will be copied in the data offset 51 * 8192 bytes.

Similarly, when Page 150 is modified, allocation of Extent 19 will be created at offset ((19 – 1) * 8 + 1) * 8192 = 145 * 8192 bytes. Page 150 will be the 6th Page in this extent and hence will be at offset 150 * 8192 bytes.. So, as you can see, these pages will be spread out in the sparse file at its own offset (depending on the number of the page). Below is a diagrammatic view of the same:

Let us modify a few rows and see what happens to the snapshot. I will update the Sales.CreditCard table. Using undocumented commands, I know that:

• CreditCardID = 615 is the first row of page 1:2550.
• CreditCardID = 719 is the first row of page 1:2551.
• CreditCardID = 2663 is the first row of page 1:2570.

I will modify Page 2550 by updating the row with CreditCardID = 615. I used the following command:

UPDATE Sales.CreditCard SET ExpMonth = 8 WHERE CreditCardID = 615

Now, lets modify Page 2551, and we will see that the Size on Disk of the Snapshot file will not change, as both of these pages belong to the same extent. However, if we modify Page 2570, which belongs to another extent, we will see that the Size on Disk of the snapshot will further increase by 64 KB (3264 KB + 64 KB = 3328 KB):

All of the experiments above clearly demonstrate the Copy On Write mechanism as we had discussed above.

Remember: Once a page has been copied over to the Snapshot, further modifications to the page will not cause it to be copied over.

This means that for a Transactional Database, each page that is modified after the Snapshot is created, will cause the page to be copied over to the Snapshot. This also means that you will see a performance degradation when a page is modified for the first time after the Snapshot is created.

If you have multiple Snapshots of the same database, each page, when dirtied, will have to be copied to each Snapshot if it does not already exists in it. To explain this further, let us assume that you have created 3 Snapshots at the following times:

• Snapshot 1 is created at 1:00 AM.
• Snapshot 2 is created at 3:00 AM.
• Snapshot 3 is created at 5:00 AM.

If Page 2500 is modified, for the first time, at 6:00 AM after say, 12:30 AM, this page will have to be copied over to all 3 Snapshots before it can be modified. On the other hand, if Page 2890 is modified at 1:30 AM, and then at 5:30 AM, the modification at 1:30 AM will cause it to be copied over to Snapshot 1; and the modification at 5:30 AM will cause it to be copied over to Snapshot 2 and Snapshot 3 (it will not be copied to Snapshot 1 as the page already exists in Snapshot 1).

Summary: If you plan to create one or more Snapshots of your Production Database, please make sure you consider the performance degradation caused by the Snapshots.

In the next post, we will investigate how data is retrieved from Database Snapshots and its effect on the Buffer Pool.

Additional Reading:

• Database Snapshots on Books OnLine.
• Misconceptions around database snapshots and transaction rollbacks.
• Database snapshots - when things go wrong.
• How It Works: SQL Server 2005 Database Snapshots (Replica).
• How It Works: Shapshot Database (Replica) Dirty Page Copy Behavior (NewPage).
• How It Works: SQL Server Sparse Files (DBCC and Snapshot Databases) Revisited.

Disclaimer: All information provided here is my personal opinion and is neither verified nor approved by Microsoft before it is published. All information, and code samples, if any, is provided "AS IS" with no warranties and confers no rights.

Let me explain the requirements first: When customers click on a Survey Link, they are redirected to a web page. This page has 20 questions; for each question, customers are supposed to provide a rating between 1 and 5, both inclusive. Of course, customers can skip any question they do not want to rate; and finally, there will be a text box where they can enter their comments. Click on Save, and the data gets saved into a database. Pretty straight-forward, isn’t it?

My customer designed the database schema such that the main tables that had the Survey Results was designed as below:

CREATE TABLE Surveys (
    SurveyID   INT       IDENTITY(1,1)  NOT NULL,
    RefNo      CHAR(10)                 NOT NULL,
    SurveyDate DATETIME                 NOT NULL,
    ...
  CONSTRAINT [PK_Surveys] PRIMARY KEY CLUSTERED 
  (    [SurveyID] ASC ),
  CONSTRAINT [CK_RefNo] UNIQUE NONCLUSTERED 
  (    [RefNo] ASC )
)
GO

CREATE TABLE SurveyResults (
    SurveyID    INT        NOT NULL,
    Answer1     SMALLINT   NULL,
    Answer2     INT        NULL,
    Answer3     INT        NULL,
    Answer4     INT        NULL,
    Answer5     INT        NULL,
    Answer6     INT        NULL,
    Answer7     INT        NULL,
    Answer8     INT        NULL,
    Answer9     INT        NULL,
    Answer10    INT        NULL,
    Answer11    INT        NULL,
    Answer12    INT        NULL,
    Answer13    INT        NULL,
    Answer14    INT        NULL,
    Answer15    INT        NULL,
    Answer16    INT        NULL,
    Answer17    INT        NULL,
    Answer18    INT        NULL,
    Answer19    INT        NULL,
    Answer20    INT        NULL )
GO

CREATE TABLE SurveyComments (
    SurveyID    INT             NOT NULL,
    Comments    VARCHAR(500)    NULL )
GO

And then, one fine day, their client asked them to design a set of new Reports; they coded the stored procedures and queries, but unfortunately, they were unable to get these queries perform as expected. Some of the requirements that were shared with me were:

• List of Surveys where customers had rated 4 or 5 in at least 10 questions.
• List of Surveys where customers had rated 2 or less in at most 5 questions.
• List of Surveys where customers had rated 5 or 1 in all questions (possibly junk surveys).
• List of Surveys where the average rating was 3 or more.

With this design, the developer had to push the data into a Temp Table and then apply all sorts of logic to the data in the Temp Table to get the desired output. This not only took a lot of time, but also increased I/O and CPU utilization. At this point, they came to me for help.

After carefully studying and analysing the situation, I came to the conclusion that with the existing design and code, further optimization was nearly impossible. We could have worked on the code, but I was not sure if we could achieve the benchmarks that were already set. I suggested a change in the schema of the SurveyResults table. My suggestion of the schema was as follows:

CREATE TABLE SurveyResults (
    SurveyID     INT        NOT NULL,
    QuestionID   SMALLINT   NOT NULL,
    Answer       SMALLINT   NULL )
GO

With this design, each Survey would be saved as 20 rows, per question. Of course, this design change warranted change in all the reporting queries; however, given the complexity of Reporting requirements, this was probably the most appropriate change. The developers made all the necessary code changes, data was transformed, and we tested, tested and tested all over again; and with all necessary Indexes in place, we were able to provide the expected performance.

This exercise, clearly demonstrated how, a schema design can affect or improve query performance.

Disclaimer: All information provided here is my personal opinion and is neither verified nor approved by Microsoft before it is published. All information, and code samples, if any, is provided "AS IS" with no warranties and confers no rights.

The biggest challenge for a SQL Server Developer is to write efficient code. The expectations are that the code must complete in the least possible time, produce correct results and use minimum resources possible. We all know that there are various ways to tune slow performing queries – Indexes (with their varieties and flavours), Query Hints, PlanGuides; you name it!

This is, however, not the intention behind this post. My intention here is to emphasize the importance of Database Design on the Query Performance. All of us understand and agree that bad code can never provide us good performance; however, we seem to overlook that good (read: Excellent!) code working on a poorly designed database can never provide us the Performance we expect. Most organizations that I have worked with, have Architect Teams and Developer Teams; however, it is unfortunate that these teams work in their own silos without much collaboration between them.

Let us consider the example we have seen in my previous post in Scenario 2. The fn_CalculateTaxPaid function takes the EmpID as input, and calculates the Tax for that employee. What is the best way to retrieve the list of employees who have paid more than 10,000 USD as Tax? Do I want to write the query as:

SELECT <Column List>
FROM <Table Name>
WHERE fn_CalculateTaxPaid (EmpID) > 10000

You might be saying – “Obviously not! You told me not to use a function on the Left of the Operator.” But then, what other options do we have?

We do have options; and this is one classic example how performance can benefit from a good database design. I can think of two ways at present, but I am sure that there will be many more ways to achieve the same.

Option 1:

Add a Computed Column to the table which stores the Tax paid by that employee. Now, create an Index on the Computed Column. Definition to add the Computed Column might look like the following:

ALTER TABLE dbo.EmpSal 
ADD EmpTAX AS dbo.fn_CalculateTaxPaid(EmpID)
PERSISTED
GO

Now, create an Index on the EmpTAX column.

CREATE NONCLUSTERED INDEX [IX_EmpTax] ON [dbo].[EmpSal] 
(
    [EmpTAX] ASC
)
GO

Now, if the original query is modified as below, can we utilize the Index and make the query faster?

SELECT <Column List>
FROM <Table Name>
WHERE [EmpTAX] > 10000

Important Note: The article Creating Indexes on Computed Columns lists all requirements for defining Indexes on Computed Columns.

Option 2:

Let us assume that due to various constraints, you are unable to add an additional column to the table. In that case, we can make use of Indexed Views. We can use the following code to create an Indexed View for our purpose:

CREATE VIEW [dbo].[vw_EmpSal]
WITH SCHEMABINDING 
AS 
SELECT [EmpID], [Salary], dbo.fn_CalculateTaxPaid(EmpID) AS [EmpTAX]
FROM dbo.EmpSal
GO
CREATE UNIQUE CLUSTERED INDEX [IX_EmpID] ON [dbo].[vw_EmpSal] 
(
    [EmpID] ASC
)
GO
CREATE NONCLUSTERED INDEX [IX_EmpSal] ON [dbo].[vw_EmpSal] 
(
    [EmpTAX] ASC
)
GO

Important Note: The Creating Indexed Views article lists all requirements for creating Indexes Views.

I hope, you can understand very well how, a great database design can influence Query Performance. A good Database Design is as important as writing good code to support your Business Requirements. There is no better ingredient for success than Great Design couples with Great Code.

Disclaimer: All information provided here is my personal opinion and is neither verified nor approved by Microsoft before it is published. All information, and code samples, if any, is provided "AS IS" with no warranties and confers no rights.

I had already written in that post the reason behind the slow performance – the EmpCountry column in the Employee table is CHAR(2) which cannot be directly compared with NCHAR(2); hence, SQL Server is forced to CONVERT the EmpCountry column values to NCHAR(2) before it can perform the comparison. Again, SQL Server does not know what the column values will look like after converting the Non-Unicode character values to Unicode. Hence, it converts each column value to Unicode and then compares the result with the value passed. Hence, the Index Scan.

What I want you to note here is the CONVERT_IMPLICIT operation that SQL Server performs; it applies this operation against each value of EmpCountry in the table. Wouldn’t it have been much easier if SQL Server was to convert the @Country parameter to CHAR(2) and then do the comparison? Well, probably… probably not. There is a huge difference between Unicode and Non-Unicode character sets; why will the designers of SQL Server take so much risk to convert the user defined value to another character set and then run into the risk of doing an inappropriate comparison?

Also, the conversion is not always so easy. Ok, lets take some examples.

Scenario 1:

Assume that the CEO of the company wants information about Employees who have joined the Organization less than an year ago. Most of the developers that I have come across, will prefer writing the query as below:

SELECT <Column List>
FROM <Table Name>
WHERE DATEDIFF(d, Join_Date, GETDATE()) < 365

Can this function be Reversed Engineered to get a Join_Date which is exactly 365 days earlier than the Current Date? Well, yes, and its very easy to do so as well.

Will SQL Server do it? No, because SQL Server has not been instructed to do so. Why will SQL Server modify the query [to: WHERE Join_Date > DATEADD(d, –365, GETDATE())]? Although it looks like an easy change considering the simplicity of the query, but in real life, queries will not be so simple. Also, such a change in the query can cause implemented Business Logic to change, and there by provide incorrect results.

The result of this query? Obviously a Table Scan will follow (considering that this is a one time query and there will be no Indexes on the Join_Date), and the query will take minutes to complete.

Scenario 2:

Assume that the Head of the Payroll Team wants information about Employees who have paid more than, say, 10,000 USD as Tax. The database has a function fn_CalculateTaxPaid that takes the Annual Salary as input and provides the amount of Tax that this accounts to. You will probably now write the query as:

SELECT <Column List>
FROM <Table Name>
WHERE fn_CalculateTaxPaid (Annual_Gross) > 10000

Think: Is there a way to Reverse Engineer the fn_CalculateTaxPaid function, and calculate the Annual_Gross that corresponds to a Tax of 10,000 USD? Probably yes, but is that easy? I do not think so.

Now, lets complicate matters a bit further. Lets say, the fn_CalculateTaxPaid function takes the EmpID as input, and calculates the Tax based on his/her Annual_Gross, Other_Income, Deduction1, Deduction2, etc. Now the Query will look like:

SELECT <Column List>
FROM <Table Name>
WHERE fn_CalculateTaxPaid (EmpID) > 10000

Now, is there a way to Reverse Engineer the function to get a list of EmpIDs that have paid a Tax more then 10,000 USD? I hope you are getting the point now.

So, now, what will SQL Server do when it encounters such a query. Obviously, a Table Scan or an Index Scan or a Clustered Index Scan will follow (depending on whether there are supporting Indexes and the type of the Indexes) and the function will be applied against each row of the table, the resulting value will be compared with the input, and the result set formed.

Conclusion:

What I am trying to explain in this post is, use of Functions, Calculations, or for that matter, any operation on the Left of the Operator will cause bad execution plans. In Scenario 1, the operator was Greater Than (>), while in the second, the operator was Less Than (<). And in both cases, we had asked SQL Server to perform an operation to the Left of the Operator, which caused the queries to choose bad execution plans. If we can avoid the use of Functions or Calculations on the Left of the Operator, we can use the existing Indexes to generate a cost effective Execution Plan, there by decreasing the overall Execution Time (which might mean shorter tea-breaks for some – like me ).

Disclaimer: All information provided here is my personal opinion and is neither verified nor approved by Microsoft before it is published. All information, and code samples, if any, is provided "AS IS" with no warranties and confers no rights.

I have seen many people confused by this question. When I say, “No, please do not expect this”, I am thrown back another question – what, then, is the purpose of having the Bulk Logged Recovery Model?

Let’s first have a look at an extract from the Books Online (http://msdn.microsoft.com/en-us/library/ms189275(v=SQL.90).aspx). Note the parts marked in Bold Red below:

This recovery model bulk logs most bulk operations. It is intended solely as an adjunct to the full recovery model. For certain large-scale bulk operations such as bulk import or index creation, switching temporarily to the bulk-logged recovery model increases performance and reduces log space consumption. Log backups are still required. Like the full recovery model, the bulk-logged recovery model retains transaction log records until after they are backed up. The trade-offs are bigger log backups and increased work-loss exposure because the bulk-logged recovery model does not support point-in-time recovery.

Hence, in short, the sole purpose of the Bulk Logged Recovery Model is to reduce Log Space Consumption by minimally logging some bulk activities, there by improving the overall performance of the bulk operations. However, this recovery model will not result in smaller log backups (smaller than in Full Recovery Model).

Before going into further details, let’s do small hands-on. We will create two databases, change their Recovery Models and create two tables in each database:

Now, we will take a Full Backup of both the databases and then a Log Backup each. After the Log Backup, we will examine the Log File Size and the Log Space Used for both the databases:

Ok. Now, we will BCP in rows into the tables in both the databases. We will BCP in the same data set in both the databases, so that after the BCP in, both the databases have identical data. I used the attached files and the following commands:

bcp FullRecoveryTest.dbo.Tab1 in D:\Tab1.txt -SSQLServerName -T -c -h "TABLOCK"
bcp FullRecoveryTest.dbo.Tab2 in D:\Tab2.txt -SSQLServerName -T -c -h "TABLOCK"
bcp BulkLoggedRecoveryTest.dbo.Tab1 in D:\Tab1.txt -SSQLServerName -T -c -h "TABLOCK"
bcp BulkLoggedRecoveryTest.dbo.Tab2 in D:\Tab2.txt -SSQLServerName -T -c -h "TABLOCK"

Now, before we take another set of Log Backups, lets examine the Log Space Usage:

We can clearly see that under the Bulk Logged Recovery Model, only 0.11474609536875 MB of Log Space has been used for the entire operation, where as under Full Recovery Model, 1.5278325428648 MB of Log Space has been used (almost 13.5 times the Log Space in Bulk Logged Recovery Model).

Does this mean that the size of the Log Backup of the BulkLoggedRecoveryTest database will be smaller than that of the FullRecoveryTest database? Let’s see:

Strange, isn’t it? The answer is again in the Books Online (http://msdn.microsoft.com/en-us/library/ms190692(v=SQL.90).aspx):

Under the bulk-logged recovery model, if a log backup covers any bulk operations, the log backup contains both log records and the data pages that were changed by bulk operations. This is necessary to capture the results of the bulk-logged operations. The incorporated data extents can make a log backup very large. Additionally, backing up the log requires access to the data files that contain the bulk-logged transactions. If any affected database file is inaccessible, the transaction log cannot be backed up and all operations committed in that log are lost.

To track the data pages, a log backup operation relies on a bulk-changes bitmap page that contains a bit for every extent. For each extent updated by a bulk-logged operation since the last log backup, the bit is set to 1 in the bitmap. The data extents are copied into the log followed by the log data. The following illustration shows how a log backup is constructed.

The proof of the above theory is in the output for the BACKUP LOG commands. For the FullRecoveryTest database, the output of the BACKUP LOG command is:

Processed 191 pages for database 'FullRecoveryTest', file 'FullRecoveryTest_log' on file 1.
BACKUP LOG successfully processed 191 pages in 0.941 seconds (1.579 MB/sec).

However, for the BulkLoggedRecoveryTest database, the output of the command is:

Processed 208 pages for database 'BulkLoggedRecoveryTest', file 'BulkLoggedRecoveryTest' on file 1.
Processed 15 pages for database 'BulkLoggedRecoveryTest', file 'BulkLoggedRecoveryTest_log' on file 1.
BACKUP LOG successfully processed 223 pages in 0.830 seconds (2.095 MB/sec).

Notice that 208 pages from the DataFile of the BulkLoggedRecoveryTest database has been backed up in the Transaction Log Backup.

Lesson learnt: We cannot expect the Transaction Log Backup Size to be smaller even if the Database is in Bulk Logged Recovery Model. This recovery model is for better Transaction Log Management, not for better Transaction Log Backup Management. Those who have Log Shipping configured, and hope to benefit from Bulk Logged Recovery model while performing maintenance tasks – this is not good news for you.

Disclaimer: All information provided here is my personal opinion and is neither verified nor approved by Microsoft before it is published. All information, and code samples, if any, is provided "AS IS" with no warranties and confers no rights.

This was probably true a couple of decades back, but we have come a long way now. With the huge advancement of technology and the computing powers being doubled probably a few hundred times, we are now not satisfied with just being able to retrieve stored data; we want to retrieve data faster, we want our operations to complete in the least amount of time possible.

All of us are aware of the fact that queries must be tuned to achieve optimal performance, some of the common ways of tuning database systems include creating appropriate indexes, updating statistics, rebuilding indexes to reduce fragmentation. However, sometimes we tend to forget that each database system is just another piece of code running on a bunch of dumb hardware; unless we design our databases properly, unless we write efficient code, we will never be able to achieve the speeds we desire. In this and subsequent posts, I will try to explain the most common mistakes that I have experienced, and how these can affect system performance.

In this post, I will try to explain the importance of appropriate data-types. To start off, let’s create a new table and populate it with data:

CREATE TABLE [dbo].[Employee](
       [EmpID] [int] IDENTITY(1,1) NOT NULL,
       [EmpFName] [varchar](50) NOT NULL,
       [EmpMName] [char](2) NULL,
       [EmpLName] [varchar](50) NULL,
       [EmpSal] [numeric](18, 2) NOT NULL,
       [EmpAddress] [varchar](250) NULL,
       [EmpCountry] [char](2) NOT NULL,
CONSTRAINT [PK_Employee] PRIMARY KEY CLUSTERED 
([EmpID] ASC))
GO

We now, insert 1,000,000 rows of data into the table. We have the following Stored Procedure:

CREATE PROCEDURE sp_GetAvgSal
    @Country NCHAR(2)
AS
    BEGIN
        SELECT AVG(EmpSal) FROM Employee WHERE EmpCountry = @Country
    END

We will execute this Stored Procedure and see how much time it takes:

SET STATISTICS IO ON
SET STATISTICS TIME ON
SET STATISTICS PROFILE ON
GO
EXEC dbo.sp_GetAvgSal @Country = 'IN'
GO

It takes ages for the query to return back the results. Finally, when it does give us back the results, here is what we get:

SQL Server Execution Times:
   CPU time = 10093 ms,  elapsed time = 282344 ms.

We can clearly see that there are lots of Reads, and we know why – there is no Index on the EmpCountry column. Hence, the Clustered Index is being Scanned for getting us back the results. Great! Since we know the source of the problem, let’s create the index to speed things up…

CREATE NONCLUSTERED INDEX [IX_Employee_Country] ON [dbo].[Employee]
   ([EmpCountry] ASC) INCLUDE ([EmpSal])

We execute the same Stored Procedure with the same parameters again. We get better results now; but not quite to our expectations.

SQL Server Execution Times:
   CPU time = 202 ms,  elapsed time = 216 ms.

What’s going on? We have a proper Index, still we are not able to achieve the performance we deserve. Let’s have a look at the execution plan. A section of the plan is:

StmtText                                                                                
----------------------------------------------------------------------------------------
SELECT AVG([EmpSal]) FROM [Employee] WHERE [EmpCountry]=@1                              
  |--Compute Scalar(DEFINE:([Expr1003]=CASE WHEN [Expr1005]=(0) THEN NULL ELSE [Expr1006
       |--Stream Aggregate(DEFINE:([Expr1005]=Count(*), [Expr1006]=SUM([Test].[dbo].[Emp
            |--Index Scan(OBJECT:([Test].[dbo].[Employee].[IX_Employee_Country])

An Index Scan? Why is the Index [IX_Employee_Country] being scanned? Shouldn’t we see an Index Seek? The answer lies in the same execution plan. Let’s have a look at that branch:

Index Scan(OBJECT:([Test].[dbo].[Employee].[IX_Employee_Country]),
   WHERE:(CONVERT_IMPLICIT(nchar(2),[Test].[dbo].[Employee].[EmpCountry],0)=[@1])) 

The EmpCountry column in the Employee table is CHAR(2) which cannot be directly compared with NCHAR(2); hence, SQL Server is forced to CONVERT the EmpCountry column values to NCHAR(2) before it can perform the comparison. Again, SQL Server does not know what the column values will look like after converting the Non-Unicode character values to Unicode. Hence, it converts each column value to Unicode and then compares the result with the value passed. Hence, the Index Scan.

How can we eliminate the Index Scan? Easy, isn’t it? Rewrite the Stored Procedure so that the @Country parameter data-type matches the data-type of the underlying table. We did this, and do you know what the result was?

SQL Server Execution Times:
   CPU time = 0 ms,  elapsed time = 1 ms.
StmtText                                                                                
----------------------------------------------------------------------------------------
SELECT AVG([EmpSal]) FROM [Employee] WHERE [EmpCountry]=@1                              
  |--Compute Scalar(DEFINE:([Expr1003]=CASE WHEN [Expr1004]=(0) THEN NULL ELSE [Expr1005
       |--Stream Aggregate(DEFINE:([Expr1004]=Count(*), [Expr1005]=SUM([Test].[dbo].[Emp
            |--Index Seek(OBJECT:([Test].[dbo].[Employee].[IX_Employee_Country])

So what did we learn? While doing comparison, you must make sure that the data-types used in the underlying tables match the data-types of the data we are comparing to. This will make sure we eliminate the IMPLICIT CONVERSIONS and thus, we can utilize the Indexes in a much better way.

Disclaimer: All information provided here is my personal opinion and is neither verified nor approved by Microsoft before it is published. All information, and code samples, if any, is provided "AS IS" with no warranties and confers no rights.

"I have Database in SQL Server, which is both Mirrored as well as Log Shipped. Log backups happen every 15 minutes; Mirroring happens in Synchronous mode and does not involve a witness. I want to break Mirroring (Remove Mirroring) and then want to re-setup the Mirroring configuration after about 30 minutes; what is the easiest way to achieve this?"

I constructed a small diagram for better understanding:

I said, "What is the size of the database?" The answer that came was, "More than 1 TB. Don't tell me you want me to start with a Full Backup; it's not feasible. It takes more than 8 hours to backup the database; and more than 6 hours to restore it, even if we use Instant File Initialization."

"Well", I said, "try restoring the first Transaction Log Backup (with NoRecovery) that was taken after the Mirroring was removed. If the Principal and Mirror were completely synchronized when Mirroring was removed (i.e., Unsent Log = 0 KB), it will get restored successfully. If they were not synchronized (Unsent Log > 0 KB), you will be prompted to restore an earlier backup. Once you get the starting point, restore all Transaction Log Backups taken in sequence (all with NoRecovery). Now reconfigure Database Mirroring again."

My friend did not believe this can be done.

Question to my readers: Do you believe this is possible? Well, give it a try…

Disclaimer: All information provided here is my personal opinion and is neither verified nor approved by Microsoft before it is published. All information, and code samples, if any, is provided "AS IS" with no warranties and confers no rights.

One such technology is the Automatic Identity Range Management. Have you ever imagined how SQL Server Replication manages the identity values of user tables? If your user table is a part of Transactional Replication with Updatable Subscriptions or the table is replicated using Merge Replication, how does Replication ensure that different values are inserted at the Publisher and the Subscribers, so that conflicts arising off using same identity values can be avoided? Let’s see...

Let’s create a table that has an identity column:

CREATE TABLE [dbo].[Orders]
(
    OrderID         INT             IDENTITY(1, 1)   NOT NULL,
    OrderDate       DATETIME                         NOT NULL,
    CustomerName    VARCHAR(100)                     NOT NULL,
    ProductName     VARCHAR(100)                     NOT NULL,
    OrderQuantity   INT                              NOT NULL,
    UnitPrice       DECIMAL(10, 2)                   NOT NULL
)

We will now publish this table using Merge Replication. We set the Publisher Identity Range as 10000 and the Subscriber Range as 1000. We will also create a subscription to this publication. For demonstration purposes, our publication will have just this one article. Also, at this point, the table is completely empty. After the initial sync, we ran the following query against both the Published Database and the Subscribed Database:

SELECT IDENT_CURRENT ('dbo.Orders')

Question: On what basis is the identity value at the Publisher and the Subscriber decided?

When we configure replication, we have to set the Publisher Identity Range and the Subscriber Identity Range. These range values are by default set to 10000 and 1000 respectively. This means that the Publisher will be assigned an identity range of 10000 (primary) + 10000 (secondary) for distribution. Of these, the Publisher will keep 1000 (primary) + 1000 (secondary) identity values for inserts on the Publisher. Among the rest, 1000 (primary) + 1000 (secondary) identity values will be assigned to the subscriber for inserts at the subscriber. The next subscriber will be assigned the next 1000 (primary) + 1000 (secondary) identity values.

Question: How does replication implement the identity values?

SQL Server Replication implements the identity values in the form of constraints that are applied on the corresponding tables. If we examine the Orders table on the Publisher, we will find a constraint by the name of repl_identity_range_<GUID> created. The GUID is based on the value of the artid column for the article in the sysmergearticles system table. The same constraint is created on the Subscriber as well. The expression for this constraint on the Publisher and the Subscriber are as follows:

Publisher:
([OrderID]>=(1) AND [OrderID]<=(1001) OR [OrderID]>(1001) AND [OrderID]<=(2001))
Subscriber:
([OrderID]>(2001) AND [OrderID]<=(3001) OR [OrderID]>(3001) AND [OrderID]<=(4001))

Question: What will happen if the current identity value at the Publisher exceeds 2001?

Once rows are entered into the Publisher Database and the Primary and the Secondary identity ranges are completely utilized, 2 new ranges are assigned to the Publisher. Hence, when the identity value on the Publisher reaches 2001, and we try to insert the 2002th value, the constraint is modified on the Publisher. The expression now becomes:
([OrderID]>(4001) AND [OrderID]<=(5001) OR [OrderID]>(5001) AND [OrderID]<=(6001))

However, the behavior on the subscriber is different. On the subscriber, if both the assigned identity ranges are completely utilized, further inserts will fail, unless synchronized with the Publisher. When the subscriber synchronizes, new identity ranges are assigned to the subscriber. If the primary range is completely utilized, during synchronization, the secondary range is made the primary range, and a new identity range is assigned, which now becomes the secondary identity range.

For SQL Server CE subscribers or subscribers running SQL Server 2000 or an earlier version, only one identity range can be assigned. Here the threshold parameter comes into play. If the identity range usage is above the threshold (default value 80%), a new identity range is assigned. Thus holes in the identity values can be created. For example, on the SQL CE subscriber, if we have the identity range set to ([OrderID]>(4001) AND [OrderID]<=(5001)), and we are currently at 4850, it means that we have utilized more than the threshold value; while synchronization, a new identity range is assigned. If now, the identity range is like ([OrderID]>(5001) AND [OrderID]<=(6001)), it means that the next identity value that will be assigned to the next new row is 5002; thus identity values between 4851 and 5001 are skipped, leaving holes in the identity values. Higher the threshold, smaller is the hole, the less fault-tolerant the system becomes.

Question: How does replication know what identity values have been assigned?

Starting SQL Server 2005, the MSmerge_identity_range system table on the Published Database has information about the current identity ranges in use. It has one row per article for the Publisher Identity Range, one row per article for the Identity Range being used by the Publisher, and one row per article for each Subscriber. The following is a screenshot of the contents of this table on the publisher:

The MSmerge_identity_range system table on the Subscribed Database, however, has one per row per article and has information about the identity range used by itself for that article.

Similar to previous versions of SQL Server, we also keep the history of all the identity range allocations in the MSmerge_identity_range_allocations system table in the distribution database. The following is a screenshot of the contents of this table:

 
(Click on the image to enlarge)

Distributing Identity Ranges to Multiple Subscribers:

Now that we know how identity rages are assigned and kept track of, lets see how the identity ranges will look like when we have multiple subscribers. Let’s assume that database PubDB of the SQL Server Instance Publisher is published and is subscribed by Subscriber1, Subscriber2, Subscriber3 and Subscriber4. The identity range allocations are as follows:

* max value for INT datatype is "2147483647".

Distributing Identity Ranges in a Re-Publishing Scenario:

In a republishing scenario, the re-publisher will also be assigned a Publisher Range, which will be used by it for distributing identity ranges to its subscribers. Lets consider the following scenario: Publisher is the main publisher, with a Publisher Identity Range of 10000 and Subscriber Identity Range of 1000, and its subscribers are Subscriber1, Subscriber2, Subscriber3, and RePublisher. The database on RePublisher is published again with a Publisher Identity Range of 7500 and Subscriber Identity Range of 500. The subscribers of RePublisher are SubSubscriber1 and SubSubscriber2.

The chart of identity range assignments will be as follows:

* max value for INT datatype is "2147483647".

So, we can clearly see that RePublisher is assigned two Publisher Identity Ranges (primary range and secondary range) and all ranges assigned to its subscribers are from these ranges only.

What if we reinitialize the Subscriptions?

If we reinitialize the subscriptions, each time we reinitialize, new identity ranges are assigned to the corresponding subscribers. This is irrespective of the fact whether the identity ranges already assigned to the subscribers are depleted. This has the potential to create holes in the identity values in the table and can even cause the assignable identity ranges to get depleted. Let’s see using an example.

I have a database that has the Orders Table, similar to the one defined earlier, with a small change. My identity column is now smallint instead of int. I have created the publication using 5000 as the Publisher Range and 2000 as the Subscriber Range. Once I create the subscriptions, I have the following identity values assigned:

Now, I reinitialize all the subscriptions. In this case, the Publisher Identity Range (1 to 2001 & 2001 to 4001) is not changed, however, each subscriber is assigned 2 new ranges. This is irrespective of the fact that I have not inserted any row in any of the databases. So, the Merge Agent starts off with the assignment:

Subscriber1: 20001 to 22001 & 22001 to 24001
Subscriber2: 24001 to 26001 & 26001 to 28001
Subscriber3: 28001 to 30001 & 30001 to 32001
Subscriber4: 32001 to 34001… Oops!!!

The max value for smallint is 32767, and the Merge Agent has tried to assign an identity range that is above the max value of smallint. This allocation will now fail, and the Merge Agent will fail with the following message:

The Publisher failed to allocate a new set of identity ranges for the 
subscription. This can occur when a Publisher or a republishing Subscriber
has run out of identity ranges to allocate to its own Subscribers or when
an identity column data type does not support an additional identity range
allocation. If a republishing Subscriber has run out of identity ranges,
synchronize the republishing Subscriber to obtain more identity ranges
before restarting the synchronization. If a Publisher runs out of identit
(Source: MSSQL_REPL, Error number: MSSQL_REPL-2147199417)
Get help: http://help/MSSQL_REPL-2147199417

What can I do to resolve this error?

To resolve this error, you can try to drop the Subscriptions and Publication, and create them once again. If the last inserted row has a low value in the identity column, this recreating everything will give you relief for some time.

However, if the last inserted identity row value is high, and close to the max value for smallint, recreating the Publication and the Subscriptions will be of no good. The only way out in such cases is to change the datatype of the Identity Column to a larger datatype (int or bigint).

Hope this helps my readers to understand the concepts of Automatic Identity Range management in Merge Replication using Microsoft SQL Server 2005 / 2008.

Disclaimer: All information provided here is my personal opinion and is neither verified nor approved by Microsoft before it is published. All information, and code samples, if any, is provided "AS IS" with no warranties and confers no rights.