Chris Skorlinski and John Sims
Microsoft SQL Server Escalation Services

If you've missed the announcement, we just published description of the Windows Azure SQL Database waits types including the SE_REPL waits.

sys.dm_db_wait_stats
https://msdn.microsoft.com/en-us/library/dn170450.aspx

Chris Skorlinski
Microsoft SQL Server Escalation Services

I want to share some tips for handling deadlocks in your SQL Database applications. You'll see the solutions for SQL Database are similar to on-premises SQL Server for which you'll well documented on web.  I'll focus on parts unique to SQL Database.

To me deadlocks are easier to understand if you create your own simple step-by-step deadlock, then see how the deadlock is shown in both SQL Server DMVs and via the SQL Database Management Portal.  Let's get started.

Generate a Deadlock

I'm using the sample database AdventureWorks.  I have 2 scripts, both doing update into the PersonPhone and Person table.  Thread 1 update PersonPhone, then Person while Thread 2 updates Person, then PersonPhone.  If these start at the same time, a classic deadlock.  You have what I need to complete my work and I have what you need to complete your work.  To make the queries less time-sensitive, I've added a WAITFOR DEALY.

Open a connection to SQL Database AdventureWorks and run script DeadLock Thread 1, then in a 2nd query window, execute Deadlock Thread 2.

--Deadlock Thread 1
BEGIN TRANSACTION
	UPDATE [Person].[PersonPhone]
		SET PhoneNumber = '999-555-1212'
	WHERE [BusinessEntityID] = 1
	
	WAITFOR DELAY '00:00:10'
	UPDATE [Person].[Person]
		SET [FirstName] = 'Chris', [LastName] = 'Skorlinski'
		WHERE [BusinessEntityID] = 1
ROLLBACK TRANSACTION

--Deadlock Thread 2
BEGIN TRANSACTION
	UPDATE [Person].[Person]
		SET [FirstName] = 'Chris', [LastName] = 'Skorlinski'
		WHERE [BusinessEntityID] = 1
	UPDATE [Person].[PersonPhone]
		SET PhoneNumber = '999-555-1212'
	WHERE [BusinessEntityID] = 1
	WAITFOR DELAY '00:00:10'
ROLLBACK TRANSACTION
/*
(1 row(s) affected)
Msg 1205, Level 13, State 51, Line 10
Transaction (Process ID 773) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
*/

Deadlock Graph

The application will receive Msg 1205 indicating that sessions\connections was "victim" and aborted while the other continued to execute until completion.  To investigate further, you can retrieve historical deadlock details by querying sys.event_log DMVs.

For this DMV, you'll need to make a connection to the MASTER database for your server.

The sys.event_log DMV retrieves various connectivity related events for SQL Database.  One of those are "deadlock" events.  Run the query below to display our recent deadlock.  The data display is retrieved from historical tables populated every few minutes.  You may need to wait a bit before the actual deadlock is displayed.

--sys.event_log
--Deadlock data displayed as XML in additonal_data
SELECT * FROM sys.event_log
WHERE database_name like 'AdventureWorks2012'
AND event_type = 'deadlock' 

The deadlock is displayed as XML data under the additional_data column.  Clicking in this column automatically opens the readable XML deadlock report.

While a bit messy, looking at the output compared to our controlled example, we can better understand the structure of this report.  The key sections are the "victim-list", the "process-list" (winner) along with details on what resources were locked and what resources were being requested. 

Victim

Winner

SQL Database Portal

You can also find deadlocks in your SQL Database portal under Databases, Administration, then Events.

You show all events or as I've done in this report filter just for Deadlocks.

Deadlock Details from SQL Database Portal

I would recommend using SQL Server Management Studio to query sys.event_log table to retrieve the deadlock details.  It's a bit more user friendly then SQL Database Portal.  If you're using the portal, here are the steps.

In upper left corner, click your database, then select Master.

Now click New Query and paste in the same sys.event_log DMV.

Here is here it gets messy. Click the |> to select the ROW displaying the deadlock.  When selected, you'll see the |> highlighted as shown below.  Then CTRL-C to copy the row and paste into your favorite text editor like NotePad or TextPad.

This is one simple example to help your improve your comfort level with deadlocks.  Here we clearly see the application was submitting 2 conflicting workload locking resources in different order leading to a deadlock.  Your deadlocks can get more complicated as multiple connections with multiple queries.  BING searching on deadlocks will provide great wealth of blog postings by SQL MVPs and white papers to help you resolve your deadlocks.

In my last blog posting I explored SQL DMVs used to find who is blocking queries from running.  By Who, I mean other queries.  We calls these "head blockers".  Our goal is to get those head blockers finished as quickly as possible to release resources for other queries.

For the "the Why" investigation we'll look at finding out what resources the head blocker is consuming, using, or waiting.  We'll use some of the same AdventureWorks2012 sample database from previous blog posting.

Most often head blocker research falls into a few common categories.

1. Head blocker is long running query\stored procedure
2. Head blocker running transactions within stored procedures and NOT releasing locks.
3. Head blocker waiting on system resources like CPU, Memory, or Disk

Long Running

Queries running a long time will hold resources preventing other queries from selecting, updating, deleting, basically getting their work done.  Examine the blocking query's "logical reads".  ( Troubleshooting Blocking in SQL Database - the who ).  Is head blocking reading more then 10,000 pages?  more then 100,000?  more then 1,000,000?  As the number of pages processed by the query increases, more time will be spent retrieving rows from disk, reading through memory, consuming CPU, etc.  Fast queries release resources quickly, less time to block. 

Compare Logical Reads of head blocker to other queries running. Are they all about the same, all equally small, all equally bad?  If head blocker has high reads, you'll want to examine the query plan and\or code logic to drop those logical reads.  I'll give more examples in future blog postings on query plans.  Searching BING for SQL MVP Query Plans you'll find great posts from SQL MVP community like Grant Fritchey, Kalen Delaney, Paul White and others on analyzing query plans

Waiting

Okay, so the head blocker is NOT running through 1 million pages.  Why is it holding pages?  Like on premises SQL Server, running SQL Database we need to examine locks.  Run the blocking example in my previous post, scroll through output and look at wait_type column.  This begins investigating into WHY query is taking a long time.  It is waiting on resources for 216 seconds!

--Show Blocked Queries
SELECT * FROM sys.dm_exec_requests
WHERE session_id > 50 and session_id <> @@spid
ORDER BY blocking_session_id -- blocking_sessions_id = 0 = not blocked

In this example its waiting on a lock.  BING search will show you details on the different wait_types. 

Locate the Head Blocker session_id

Next query sys.dm_tran_locks to see all the SQL system resources that are locked by this head blocker.  Look through the output to get an understanding of what resources is the head blocker holding.  One or many tables?  One or many pages?

--Show LOCKS for blocking and blocked
SELECT request_session_id, request_status,resource_type, resource_description, 
	resource_associated_entity_id, request_mode, request_status
FROM sys.dm_tran_locks 
WHERE request_owner_type = N'TRANSACTION' 
    AND (request_session_id = 317 -- Head Blocker
	OR request_session_id = 878)  -- Being Blocked
ORDER BY request_session_id
GO

This query shows both resources GRANT and WAIT.  How many rows are held by the GRANT?

Using the Partition_ID we can find out table name for the WAIT resource.

-- resource_associated_entity_id
SELECT object_name(object_id) 'Table Name',
	object_id, index_id, row_count, in_row_data_page_count, lob_reserved_page_count
    FROM sys.dm_db_partition_stats
    WHERE partition_id=72057597391601664

Table Name        object_id   index_id    row_count    in_row_data_page_count lob_reserved_page_count
----------------- ----------- ----------- ------------ ---------------------- -----------------------
Person            997578592   1           19972        2747                   0

Results here show waiting on the Person table, index #1.  Now compare the query or Stored Procedure code for both the head blocker and the blocked to see where they overlap on the Person table.  Maybe you can tune one query to run faster reducing the blocking time, or change the programming logic. to query then release the table.

Release Locks

Stored procedures coded to release locks reduce the opportunity for blocking.  I've seen many procedures where code written to hold the table locks longer then needed.  For example:

Begin Transaction
SELECT a table 1
UPDATE a table 1
SELECT table 2
UPDATE table 2
Commit Transaction

If the 2nd SELECT runs longer then expected, the UPDATE LOCKS on the first table will not be released.  Another user will be blocked waiting for Table 1 while problem is the long running SELECT on Table 2.  Like my dog Roscoe, you'll be chasing your tail all day long.

Using No Locks not always the answer.  Try coding stored procedures to be as lean as possible.  Review the Isolation Levels, load data into TempDB or table variables.  When developing, add WAITFOR DELAY '00:05:00' as last line of the SP then examine sys.dm_tran_locks to learn what is locked when the stored procedure ends.  Any thing unexpected in the list?

Waiting on System Resources

If the head blocker is waiting on system resources you may need to scale out your SQL Database deployment, but first look at your overall work load.  Do you have other queries consuming large resources?  Who is the highest logical reader?  One or two queries with high logical reads will load up buffer pool and tie up worker threads needed by other queries.  A query updating a million rows which need to be committed on your failover secondary servers could be causing SE_REPL waits on other queries.

Start by polling sys.dm_exec_sessions and sys.dm_exec_requests over time looking at physical writes and logical reads.  Here are couple queries to help.

-- Stuart Ozer, Connor Cunningham, Chris Skorlinski (revised)
-- View Currently Executing (dm_exec_requests)
--		wait_type, wait_resource, statement_text, statement_plan
--		execute user-defined statement_level_query_plan in the current DB.
select pln.*, req.* from sys.dm_exec_requests as req
CROSS APPLY statement_level_query_plan(plan_handle) as pln
where session_id <> @@spid AND statement_text like
'%' +
replace(
left(
    substring((select text from sys.dm_exec_sql_text(sql_handle)), 
            statement_start_offset/2, 
            1+      case when statement_end_offset = -1 
                    then LEN((select text from sys.dm_exec_sql_text(sql_handle))) - statement_start_offset/2
                    else statement_end_offset/2 - statement_start_offset/2 
                    end) 
        ,3000)
, '[','[[]') + '%'

--Written by: Jonathan Kehayias SQLskills.com MVP
--Modified by: Chris Skorlinski, Microsoft SQL Server Escalation Services
select dm_tran_locks.request_status,
       dm_tran_locks.request_session_id, 
            dm_tran_locks.resource_database_id,
            db_name(dm_tran_locks.resource_database_id) as dbname,
            CASE 
                  WHEN resource_type = 'object'
                        THEN object_name(dm_tran_locks.resource_associated_entity_id)
                  ELSE object_name(partitions.object_id)
            END as ObjectName,
            partitions.index_id,
            indexes.name as index_name,
            dm_tran_locks.resource_type, 
            dm_tran_locks.resource_description, 
            dm_tran_locks.resource_associated_entity_id, 
            dm_tran_locks.request_mode
from sys.dm_tran_locks
left join sys.dm_db_partition_stats partitions on partitions.partition_id = dm_tran_locks.resource_associated_entity_id
join sys.indexes on indexes.object_id = partitions.object_id and indexes.index_id = partitions.index_id
where resource_associated_entity_id > 0
  and resource_database_id = db_id()
order by request_session_id, resource_associated_entity_id

You'll find more details on these in My Links blog posting.

In this blog I’ll walk through troubleshooting blocking in SQL Database.  As you’ll see, we’ll many of the same techniques used for on-premises SQL Servers with one exception, will not have use of SQL Server Profiler to capture server activity.  While Profiler is often used, started with SQL Server 2005, the SQL DMVs provide the details about blocking.

For blocking, that is one user is holding resources need by one or more users, first step is to identify the “who”, that is head blocker and see what resources they hold.  Next is understanding “why” they are holding resources followed by “what” option do we have to solve this problem.  I’ll cover the “who” in this posting.

Start by making 2 connection to SQL Database. Execute the 2 scripts in each query.  These examples are using the AdventureWorks2012 sample we installed earlier.

--Query 1: Start UPDATE blocking
BEGIN TRANSACTION
	UPDATE [Person].[Person] WITH (holdlock)
		SET [FirstName] = 'Chris', [LastName] = 'Skorlinski'
		WHERE [BusinessEntityID] = 1
	WAITFOR DELAY '00:04:00'
ROLLBACK TRANSACTION
 
--Query 2: Being Blocked
BEGIN TRANSACTION
	UPDATE [Person].[Person]
		SET [FirstName] = 'Chris', [LastName] = 'Skorlinski'
		WHERE [BusinessEntityID] = 1
ROLLBACK TRANSACTION

In this example Query 1 performs update using a "holdlock", okay not most elegant, but it works, then goes into a 4 minute WAIT before rolling back the transactions.  If you need more time while exploring the DMVs, just re-run these queries.

Now, in a 3rd query window connect again to same AdvenstureWorks2012 and executes queries below.

SELECT * 
FROM sys.sysprocesses
SP_WHO
SP_WHO2
 

Sorry, those don't work, you can query dm_exec_session and dm_exec_requests to look at blocking details.

--Who's in my database
SELECT host_name, login_name, login_time,program_name,status,cpu_time, logical_reads
FROM sys.dm_exec_sessions
WHERE status <> 'sleeping' AND session_id <> @@spid
--Show work being performed, blocking, waits, open_transactions, query_hash, query_plan_hash
SELECT session_id,start_time,status,logical_reads,cpu_time,total_elapsed_time,
	command,blocking_session_id,wait_type,wait_time,last_wait_type,wait_resource,connection_id, 
	sql_handle,plan_handle,query_hash,query_plan_hash
FROM sys.dm_exec_requests
WHERE session_id <> @@spid
ORDER BY blocking_session_id -- blocking_sessions_id = 0 = not blocked

From dm_exec_requests look at the blocking_session_id column.  Values = 0 indicate no blocking and often are the "head blockers".  For my example, you'll see query 1, "running" with command "WAITFOR", and wait_type "WAITFOR", normally you'll find more normal SQL commands such as an UPDATE, DELETE, INSERT.  For our WAITER, you can see "suspended" with wait_type LCK_M_U, an update lock request.  These guys aren't going anywhere real fast.

When troubleshooting your blocking, look closer at the logical_reads column, high values here (10K+) often indicate a busy query consuming too many resources and focus for tuning opportunities.  I'll cover this more in another post.

Identifying WHO also allows us to retrieve the sql_handle and plan_handle from dm_exec_requests.  We can use built-in system functions to retrieve SQL statement and query plans for queries shown.

--Retrive XML Query Plan for the UPDATE statement
SELECT * FROM sys.dm_exec_query_plan(0x06001F00C5B3ED0EE0FE084D0400000001000000000000000000000000000000000000000000000000000000)

Results is the query_plan you can click-open in SQL Server Management Studio.  Again, more on query plans later.

If you search the web, you'll find more example of using dm_exec_requests.  Try some of these queries against our SQL Database AdventureWorks2012 example.  Here is one I pull then modified from Stuart Ozer and Connor Cunningham (link in my "favorite links" posting I did earlier.

-- Stuart Ozer, Connor Cunningham, Chris Skorlinski (revised)
-- View Currently Executing (dm_exec_requests)
--		wait_type, wait_resource, statement_text, statement_plan
select pln.*, req.* from sys.dm_exec_requests as req
CROSS APPLY statement_level_query_plan(plan_handle) as pln
where statement_text like
'%' +
replace(
left(
    substring((select text from sys.dm_exec_sql_text(sql_handle)), 
            statement_start_offset/2, 
            1+      case when statement_end_offset = -1 
                    then LEN((select text from sys.dm_exec_sql_text(sql_handle))) - statement_start_offset/2
                    else statement_end_offset/2 - statement_start_offset/2 
                    end) 
        ,3000)
, '[','[[]') + '%'

In my next posting I'll talk about WHY they may be blocking and WHAT are your next steps to prevent blocking.

I’ll be posting SQL Database (Windows Azure SQL Database) performance troubleshooting tips over the next few weeks.  For most of these examples I’ll use the AdeventureWorks2012 sample database available for free download on CodePlex.

AdventureWorks Community Samples Databases for Azure SQL Database
https://msftdbprodsamples.codeplex.com/releases/view/37304

Once you download, execute from a CMD prompt to run the install script against your SQL Database.  You can also specify a “local” SQL Server if you want the same database installed both on-premises and in cloud.

CreateAdventureWorksForSQLAzure.cmd {servername} {user} {pw}

Be sure to use the fully qualified SQL Database server name and administrative user as in my example below:

CreateAdventureWorksForSQLAzure.cmd 'abcxyz.database.windows.net' 'administrator_user' 'administrator_password'

Once the database is created you can connect using SQL Server Management Studio and browse similar to on-premise database.

Under the “Options >>” button you can switch the connection between the AdventureWorks2012 and MASTER databases.

SELECT * FROM sys.dm_db_wait_stats
ORDER BY wait_time_ms desc

wait_type                                     waiting_tasks_count  wait_time_ms         max_wait_time_ms     signal_wait_time_ms
--------------------------------------------- -------------------- -------------------- -------------------- --------------------
LCK_M_U                                       7                    640004               225320               0
ASYNC_NETWORK_IO                              12508                62532                192                  345
PAGEIOLATCH_SH                                1100                 10999                1362                 69
SE_REPL_COMMIT_ACK                            2029                 7551                 244                  372
SE_REPL_SLOW_SECONDARY_THROTTLE               30                   7356                 1178                 10
SOS_SCHEDULER_YIELD                           8915                 1799                 15                   1778

Chris Skorlinski
Microsoft SQL Server Escalation Services

Tools

CSS SQL Azure Diagnostics tool released
--Web Portal tool to display historical query performance stats from SQL Database

Tools and Utilities Support (Windows Azure SQL Database)

SQL Azure CAT team article on loading data into SQL Azure
-- Other AzureCAT content can be found on:  Windows Azure Customer Advisory Team

Sync Framework Programming Tools to move data between On-Premise SQL Server and cloud SQL Database

Blog Postings

Main Microsoft Windows Azure blog along with SQL Database posts

Gaining Performance Insight into Windows Azure SQL Database
--Wayne Walter Berry posting on displaying SQL Database Query plans

Improving Your I/O Performance
--Wayne Walter Berry posing on Query plans and covering indexes

Introducing sys.database_connection_stats for Troubleshooting Windows Azure SQL Database
--Yes, another Wayne favorite of mine.

SQL Database Wait Events
--I follow all of Thomas Larock BLOG postings.  He's writing style is easy to read and informative.

New BLOG on troubleshooting and best practices for SQL Database.  This one has great potential.

MSDN White Papers

Windows Azure SQL Database and SQL Server -- Performance and Scalability Compared and Contrasted
--On my “must read” list of Conor Cunningham papers.

Windows Azure SQL Database Connection Management
--Another “must read” by Selcin Turkarslan and Srini Acharya.  This coverings throttling and reconnects is great detail.

Best Practices for the Design of Large-Scale Services on Windows Azure Cloud Services
--Another great article providing best practice insight into scaling out Windows Azure and SQL Database.  If you’re just using SQL Database, skip down to the SQL Database section, it alone is a great technical overview of SQL Database