While troubleshooting some intermittent deadlocking in one of my databases I was surprised to learn that both commands were deleting single rows from different tables and both lookups were on indexed values. The commands were table scanning and deadlocking on a seemingly unrelated resource in another table.

After digging in I discovered a chain of triggers and foreign key references with cascade deletes leading back to the deadlocked table, but it still didn't answer why they were deadlocking in the first place and needed a bit more investigation.

Here's what was happening using an example table pair:

Possible Actions on the Order Table

1. Insert a record: No problem, just a normal singleton insert
2. Update a record: No problem if we are not modifying the value for Id. If we are, then I'd question the underlying database design
3. Deleting a record: Here's the problem. Deleting an Order means that SQL needs to check for matching values in OrderItem.OrderID and delete them (cascade delete), or raise an error if it does.

If there's no index on OrderId, then SQL must do a table scan on OrderItem every time we delete a row from Order which takes time and can do deadlock-happy activity like taking range X-X locks . The simple solution to this is creating a nonclustered index on OrderId:

CREATE NONCLUSTERED INDEX NCI_OrderITem_OrderID
ON OrderItem(OrderId)

This solves it but raises the question of how many other referencing columns have the same problem. I created a query to look for columns that reference other tables and wasn't the first column in an index. I didn't want the overhead of maintaining a indexes that would never be used, so I restricted it to reference only tables with delete activity. This is what I came up with:

Finding Missing Foreign Key Indexes

--Run this from the database you are interested in
--Each delete shown in the output represents a table scan on the referencing table

WITH ic AS (SELECT sic.object_id, sic.column_id
FROM sys.index_columns sic
WHERE key_ordinal=1
)
SELECT object_name(fkc.parent_object_id) 'Referencing_table',
sc.name 'Referencing_Column',
object_name(fkc.referenced_object_id) 'Referenced_Table',
object_name(fkc.constraint_object_id) 'FK_Constraint_Name',
ios.leaf_delete_count,ios.nonleaf_delete_count
from sys.foreign_key_columns fkc
JOIN sys.dm_db_index_operational_stats (db_id(), NULL, NULL, NULL) ios ON (fkc.referenced_object_id = ios.object_id)
LEFT JOIN ic ON (fkc.parent_object_id = ic.object_id AND fkc.parent_column_id = ic.column_id)
JOIN sys.columns sc ON (fkc.parent_object_id = sc.object_id AND fkc.parent_column_id = sc.column_id)
where 1=1
AND ic.object_id IS NULL
AND ios.index_id in (0,1)
AND (ios.leaf_delete_count > 0 OR ios.nonleaf_delete_count > 0)

What the query returns

We get a result set similar to the following (*two columns omitted):

Conclusion

Over 5 million deletes from order and with each causing a table scan on OrderItem, and 665000 deletes from customers each causing a table scan on Profile regardless of whether the delete even cascaded.

The 74 deletes from Catalog cause the same problem, but here we get into the question of whether the overhead of maintaining an additional 5 indexes would be worth the benefit. For Profile and OrderItem it was a no-brainer to add nonclustered indexes to the CustomerId and ItemId columns.

Synchronize From Backup

This method cannot be done from the GUI and must be done through script. The basic process involves making a backup of the publisher database, restoring it to the subscriber, then create the subscription specifying the backup file. SQL will read the Log Sequence Number (lsn) from the header and apply all transactions that have occurred since that point.

Advantages

• Much faster than automatic for large databases
• Much faster than automatic for long distances
• No blocking on the publisher database

Disadvantages

• Restoring the database overwrites the current database on the subscriber
• Must manually correct any incompatible features on the subscriber
• Database permissions are replaces with the publisher permissions
• Orphaned users

Best Used When

• Database is very large
• Publisher and subscriber are geographically distant over a WAN
• Most or all of the database is published
• Publisher schema works as a subscriber schema with little modification

Notes

1. Must use a publisher backup: One of the caveats to sync with backup is that the backup must be from the publisher database. You cannot take a backup from another subscriber and sync with it, though there are some tricks to make it work.
2. Database overwritten: Since the entire database is overwritten, all objects in it are replaced with whatever was on the publisher database. The usual fix for this is to script everything on the subscriber that is needed and replace them after restoring the backup. Some items include:
   1. Permissions
   2. Roles
   3. Users
   4. Objects (functions, stored procedures, etc)
   5. Indexes
3. Orphaned SQL Users: Any SQL logins created with a different SID won't map to user in the database when restored, and you'll see errors like this:

   Error: 18456, Severity: 14, State: 38. Login failed for user 'foo'.
   Reason: Failed to open the explicitly specified database. [CLIENT: x.x.x.x] 

   The ways to fix these is to recreate the login using the SID from the publisher, or remap the user to the login using either sp_change_users_login or ALTER USER

4. Not For Replication Option: As a general rule all objects in the publisher database that can have the 'NOT FOR REPLICATION' option should be created with it. What the option does is ignore the specific feature if a replication agent is doing the work. There are four of these:
   1. Foreign Keys: The constraint was already checked on the publisher and does not need to be rechecked on an identical subscriber.
   2. Check Constraints: Same reason as foreign key
   3. Identity Columns: The publisher will insert the identity value directly into the table to maintain the same values between both servers, so the replication agent would encounter an insert error if not set. One side effect of this is that the identity value will not update, so if inserts need to be made on the subscriber (peer-to-peer replication, for example) then update the identity seed using a range that won't conflict with the publisher using DBCC CHECKIDENT (RESEED)
   4. Triggers: Generally it's better to publish the base tables that a trigger modifies instead of letting it fire on the subscriber. Replicating both the trigger with 'for replication' and the base table will cause a conflict.
5. Long Copy / Restore Time: With extremely large databases it can take several hours to copy it across the network and restore. The way to work around this problem is to do the initial restore with the NORECOVERY option, then create and apply a differential backup to catch up on recent transactions.
6. Required transactions missing: When you try to create the subscription you get the message below.

Msg 21397
The transactions required for synchronizing the nosync subscription created from the specified backup are unavailable at the Distributor. Retry the operation again with a more up-to-date log, differential, or full database backup.

This is usually caused by not having a minimum retention time set on the distributor and the distribution cleanup agent removes the transactions since they have already been distributed to all existing subscribers. The solution is to disable the distribution cleanup agent before taking the backup. 

If you already see the message it's too late and the best way out is to restore the database again with NORECOVERY, then disable the distribution cleanup agent, and finally backup a differential and restore it. Use the differential to synchronize.

 
/*===============================================================================================*/
/*Add Transactional snapshot subscription */
/*Restore and synchronize using backup */
/*SQL 2005 or SQL 2008 */
/* */
/*Use Ctrl-Shift-<M> to replace template parameters */
 
/*=============================================================================================================================*/
--1. Run on Distributor: <Distributor, varchar, DistServer>
--Stop distribution cleanup agent
--
--You'll see an error if the job isn't running, it's normal
USE <Distribution_DB, varchar, distribution>
GO
EXEC msdb..sp_update_job @job_name = 'Distribution clean up: <Distribution_DB, varchar, distribution>', @enabled = 0;
EXEC msdb..sp_stop_job @job_name = 'Distribution clean up: <Distribution_DB, varchar, distribution>'

/*===============================================================================================*/
--2. Run on Publisher: <Publisher, varchar, PubServer>
--In the publisher DB: <PublisherDB, varchar, PubDB>
--
--Steps 1 and 2 manually delete the subscription from the publisher and subscriber.
--If you drop the subscription through the GUI, and select the option to connect to
--the subscriber and delete the record, you can skip steps 1 and 2.
--This way you don't have to bother with opening the new query window for step 2

Summary

Initialization from backup solves a lot of problems when trying to synchronize large databases but creates a few of its own. These problems can be mitigated with some planning and hopefully this template helps to simplify the process.

Automatic Synchronization

This is the default scenario and the one you get if you create it using the UI.

Advantages

• Can be done entirely using the GUI
• Easiest to configure
• Only copies data involved with the publication
• Least amount of administrative effort

Disadvantages

• Extremely slow, especially over long distances
• Causes blocking on the publisher
• Requires snapshot space on the distributor
• Loss of permissions on subscriber tables

Best used when

• Publication is small
• Synchronization time is unimportant
• Publisher, Distributor, and Subscriber in the same datacenter
• No long-running transactions on published tables during snapshot generation

Notes

1. Blocking: Contrary to popular belief SQL server does take a Sch-M (Schema Modification) lock on published tables even with concurrent snapshot processing. What this means is that if you have some long-running selects on the table it's going to wait for those to finish before it gets its lock, and every statement behind it will be blocked as well.

2. Distributor Space: The main issues with large data sets involve generating the snapshot and storing it on the distributor. One thing that often happens is with multiple failed attempts the server might have several snapshots and fill the drive. If the distributor has sufficient space it might work reasonably well if the machines are geographically close together.

3. Latency: Long geographic distances have the most profound effect where subscriptions of only a few GB in size can take days to synchronize. Pull subscriptions generally perform better in this environment but it will still take a while to apply the snapshot and comes at the expense of having to manage replication jobs on multiple servers.

4. Permission Issues: Another concern with the snapshot process is permission loss. The default configuration for a publication is to drop and recreate the destination table which also removes any permissions that were on it. The most common solutions to this are to configure the published objects to truncate rather than drop and create, or configure a post snapshot script that applies any table-level permissions.

Initialization Process

The best part is that all work is done on the publisher and it's as simple as running sp_addsubscription. I still recommend a script even when the GUI is available especially when adding the second subscriber because there's less chance of misconfiguring, or mismatching the current subscription. Unless I'm trying to do something special with a new subscriber I create them all from a template.

The first two steps in this template are for manually deleting the existing subscription. If you do this step through the GUI, you can skip the hassle of switching connections to delete the subscriber record.

/*===============================================================================================*/
/*Add Transactional snapshot subscription           */
/*Synchronize using snapshot                        */
/*SQL 2005 or SQL 2008                              */
/*                                                  */
/*Use Ctrl-Shift-<M> to replace template parameters */
 
/*===============================================================================================*/
--1. Run on Publisher: <Publisher, varchar, PubServer>
--In the publisher DB: <PublisherDB, varchar, PubDB>
--
--Steps 1 and 2 manually delete the subscription from the publisher and subscriber.
--If you drop the subscription through the GUI, and select the option to connect to
--the subscriber and delete the record, you can skip steps 1 and 2.
--This way you don't have to bother with opening the new query window for step 2
 
USE <PublisherDB, varchar, PubDB>
GO
 
--Drop the existing subscription if present
--
IF EXISTS (
  SELECT 1
  FROM syspublications sp
  JOIN sysarticles sa ON (sp.pubid = sa.pubid)
  JOIN syssubscriptions ss ON (sa.artid = ss.artid)
  WHERE sp.Name = N'<Publication, varchar, PubName>'
  AND  ss.srvname = N'<NewSubscriber, varchar, NewSubServer>'
  AND  ss.dest_db = N'<NewSubDB, varchar, PubDB>'
)
  EXEC sp_dropsubscription
        @publication = N'<Publication, varchar, PubName>',
        @subscriber = N'<NewSubscriber, varchar, NewSubServer>',
        @destination_db = N'<NewSubDB, varchar, PubDB>',
        @article='all'
 
/*===============================================================================================*/
--2. Run on Subscriber: <NewSubscriber, varchar, NewSubServer>
--In the Subscription DB: <NewSubDB, varchar, PubDB>
--
--You can skip this step if you deleted the subscription record from the GUI
USE <NewSubDB, varchar, PubDB>
GO
 
  EXEC sp_subscription_cleanup
        @publication = N'<Publication, varchar, PubName>',
        @publisher_db = N'<PublisherDB, varchar, PubDB>',
        @publisher = N'<Publisher, varchar, PubServer>'
 
/*===============================================================================================*/
--1. Run on Publisher: <Publisher, varchar, PubServer>
--In the publisher DB: <PublisherDB, varchar, PubDB>
USE <PublisherDB, varchar, PubDB>
GO
 
exec sp_addsubscription
  @publication = N'<Publication, varchar, PubName>',
  @subscriber = N'<NewSubscriber, varchar, NewSubServer>',
  @destination_db = N'<NewSubDB, varchar, PubDB>',
  @subscription_type = N'Push',
  @sync_type = N'automatic',
  @update_mode = N'read only',
  @subscriber_type = 0
 
exec sp_addpushsubscription_agent
  @publication = N'<Publication, varchar, PubName>',
  @subscriber = N'<NewSubscriber, varchar, NewSubServer>',
  @subscriber_db = N'<NewSubDB, varchar, PubDB>',
  @job_login = null,
  @job_password = null,
  @subscriber_security_mode = 1,
  @frequency_type = 64,
  @frequency_interval = 1,
  @frequency_relative_interval = 1,
  @frequency_recurrence_factor = 0,
  @frequency_subday = 4,
  @frequency_subday_interval = 5,
  @active_start_time_of_day = 0,
  @active_end_time_of_day = 235959,
  @active_start_date = 0,
  @active_end_date = 0,
  @dts_package_location = N'Distributor'
 
--Start the snapshot agent to start synchronizing the subscription
exec sp_startpublication_snapshot @publication = N'<Publication, varchar, PubName>'
 /*===============================================================================================*/

Summary

Hopefully this helps automate some of the subscription setup process and avoid some of the more common pitfalls with using the Automatic synchronization method

Initializing transactional replication for the first time or reinitializing when it goes out of sync can be frustrating but knowledge and planning can take most of the pain out of it. This guide is designed for SQL Server database administrators to synchronize transactional replication subscriptions quickly and efficiently. It requires a working knowledge of replication and administration concepts including:

• Database Backup and Restore
  • Full, Differential, and Transaction logs
• Replication Concepts
  • Publisher, Distributor, and Subscribers
  • Publications and Subscriptions
  • Publication and subscription options
  • Replication Agents and jobs

If you aren't sure about these concepts I'd recommend creating a couple of regular transactional replication push publications on a non-production machine with read-only subscriptions and play around with the different settings and agent jobs.

Most scenarios covered involve push replication with read-only subscribers because it tends to be the most common setup. They should work similarly for peer-to-peer and updating subscribers but would need to be updated accordingly.

Background

The crux of the synchronization problem boils down to two main tasks:

1. Getting the initial data to the subscriber
2. Creating the subscription metadata in the publishing, distribution, and subscription databases

The glue that binds the process together is the log sequence number (lsn) from the publisher database and this is how SQL server keeps track of where it's at. I tend to make a lot of references to time and what I usually mean is at a specific lsn at the publisher database.

In addition to these requirements the process should meet secondary objectives:

1. Minimize the complexity involved with synchronization
2. Minimize disruptive activity on the publisher
3. Minimize impact to the business environment
4. Minimize the time needed to synchronize

Making a mistake during initialization tends to punish harshly, typically with a lot of late nights and weekends figuring out what went wrong, how to fix it, and explaining it to downstream customers.

Getting the initial dataset to the subscriber

This process starts by getting a consistent snapshot of all of the articles from the same point in time at the publisher and applying it to the new subscriber. Usually this is done by the snapshot agent or from a database backup. It could also be done by directly copying tables but this tends to be time consuming and also introduces the problem of data consistency since you can synchronize the entire process to a specific lsn. There's a possibility of using a backup from a different database (perhaps another subscriber), but we can't use the lsn from that file since lsns are specific to an instance of a database and we require the lsn from the publisher.

Just as important is ensuring that transactions after the initial lsn and before the initial synchronization are not lost. The worst part about this is that we usually don't discover this until we try to create the subscription and get a message that looks something like this:

Msg 21397
The transactions required for synchronizing the nosync subscription created from the specified backup are unavailable at the Distributor. Retry the operation again with a more up-to-date log, differential, or full database backup.

The usual suspects for missing data in this time frame are: 

1. Publication doesn't exist: The publication needs to exist so that there is a start point for synchronization, and it needs to have existed before the time the initial data snapshot is taken.
2. Distribution Cleanup Agent: This is the job that runs on the distributor and dutifully cleans up transactions that have already been applied at all subscribers. The problem occurs because the minimum transaction retention period is set to 0 by default. Since the subscription usually does not yet exist while copying the data, it removes the transactions as soon as all of the other subscribers have applied them. You can disable this job during the process, or you can set a minimum retention time.  

The number of commands between the lsn of the backup and the current lsn at first synchronization is critical because all of that information needs to be stored at the distributor until it's applied. The more commands waiting the worse the performance will be when SQL reads, applies, and then cleans up this chunk of data. This is why it's important to minimize the synchronization time as much as possible. This is also a reason against having a large minimum retention time.

Since we can synchronize from any backup file including a transaction log backup, the easiest trick to minimize the time lag is to restore differential and logs to bring the database as close to the current time as possible. The other bonus is that these operations are largely parallelizable since we can copy the diff and log files at the same time the full is restoring.

A second important trick to minimizing time lag is to create scripts while the data is copying / restoring and have them ready to execute when the database is restored.

Creating the publication and subscriptions

Implementing a subscription with any kind of nontrivial synchronization plan requires a lot of verbose scripting and even simple implementations should be scripted for disaster recovery and maintenance purposes. One easy way to do this is by creating something close to what you want in the GUI and using 'Generate Scripts...' on the publication to make scripts which are then modify to a scenario. Keep the entire script for disaster recovery but for the purposes of syncing a new subscriber we are generally only interested in the last two lines:

exec sp_addsubscription @publication = N'MyPub', @subscriber = N'Subserver'[...]
exec sp_addpushsubscription_agent @publication = N' MyPub', @subscriber = [...]

Another way is to create templates that support a specific scenario, then use the 'Replace Parameters' function to fill in the Publisher, Subscribers, etc. In addition to saving time these templates reduce the odds of forgetting a critical step during the process which usually results in starting over from the beginning.

The time between determining the initial lsn and creating the subscription is critical because modifications during this period can cause SQL to reject an lsn as unsuitable. Furthermore, starting or stopping replication agents during the process can cause data loss depending on the specific scenario. Being unsure about whether the sync process lost data is often worse than just choosing different sync method because validating that two large distributed data sets are the same can be costly.

SQL 2005 allows no-sync types of 'replication support only' and 'initialize with backup' which limits our options for lsns we can start from. 'Replication Support Only' requires that publisher and subscriber have identical data (i.e. quiesced). 'Initialize with Backup' allows us to start from a different lsn, but not any arbitrary one. SQL 2008 allows option 'initialize from lsn' which does allow any arbitrary valid lsn to be used.

A tempting workaround for the SQL 2005 issue is to create a subscription with a different lsn, and then immediately update it before first synchronization with something like:

EXEC sp_setsubscriptionxactseqno 'Publisher', 'PubDB', 'Publication', <My_arbitrary_lsn>

The difficulty with this approach is that the distribution agent is what sets up the subscription, after which it immediately begins synchronizing. If a method depends on starting exactly from this lsn then there is a risk of introducing data inconsistencies for records applied between the original lsn and the updated one. 

It might be possible to try manipulating the lsn by creating the initial subscription with frequency_type = 2 (on demand) so that it doesn't automatically start and then change the subscription_seqno number in the MSSubscriptions table on the distributor. Unfortunately, every experiment I've tried with this has resulted in lost transactions so I wouldn't recommend it, but there are some other ways to work around the problem.

Summary

Synchronizing transactional replication subscriptions involves getting all of the replicated tables to the subscriber with the least amount of effort and disruption. Once everything is in place it becomes a matter of creating the subscription and letting SQL know which lsn it should start processing from.

In the coming weeks I will post info about common replication scenarios and some methods for doing them.