Introduction

Often large amounts of data need to be quickly loaded into a database. A common approach is to fill a DataTable and use the SqlBulkCopy class to load the data. The problem with this approach is that the data set must be materialized in memory. This is inefficient because the data must be copied and transformed several times and the cache utilization is poor. As well it does not scale because memory is usually a limited resource. But it is far faster than one row at a time inserts.

An alternative is to implement an IDataReader and use the SqlBulkCopy class to load the data but the implementation requirements are poorly documented. I have developed the BulkDataReader class that makes the implementation straightforward. With the class, it is possible to stream the data from one source into a database. Since the data has fewer transformations and is streamed from buffer to buffer (and thus utilizes caches better) the performance and resource utilization is much better than other approaches. In practice, the limiting factor with this approach is how fast the data can be read from the data source. In early releases of the .Net Framework, the DataTable approach was much slower than the IDataReader approach but DataTables are now much more efficient and the approaches have comparable performance when memory is not a constraint.

Bulk loading data is much faster that loading data with single inserts because the repeated overhead of transferring the data, parsing the insert statement, running the statement and issuing a transaction record is avoided. Instead, a more efficient path is used into the storage engine to stream the data. The setup cost of this path is however much higher than a single insert statement. The break-even point is typically around 10 to 100 rows. Once the data exceeds this size, bulk loading is almost always more efficient.

The choice of clustering key, index fill factor and compression has an impact on the time to load large data sets. In general, integer based surrogate keys and sequence objects are the best choices for clustering key. Integer based keys are small and new rows are inserted at the end of the clustered index. This means that SQL Server’s lazy writer can write the page once when it is full and more rows fit in a page. By contrast, GUID based and natural keys are inserted non-sequentially and are larger so they require more page writes for the same amount of data. Also, they can cause page splits, which reduces query performance. To minimize the effect of page splits in non-sequential keys, index fill factors should be set to allow some space for growth in index pages (80% is usually a reasonable value). Compression adds CPU overhead to reading and writing data but it often improves overall performance by increasing the number of rows per page and thus reducing the amount of I/O required.

Often a heap is used to stage data before inserting it into the target table. This consumes more disk bandwidth but has the advantage that the data loading can be buffered and merges are possible. Typically, a stored procedure runs continuously merging the top n rows into the target table and deleting them from the staging table. Heaps work well for this scenario since enqueue and dequeue operations have low cost.

Table partitioning when combined with Windows Server 2012 Storage Spaces can provide a large increase in physical storage bandwidth. But that is too large a topic for this posting. In general, good performance

BulkDataReader

The download has all the code and test code for the BulkDataReader class. To implement an IDataReader using the BulkDataReader, do the following:

·         Override the SchemaName and TableName properties with meaningful names. These need not match the target schema and table names and are useful for debugging.

·         Override the AddSchemaTableRows method to declare the schema of the source information. In general, this should match the target schema but some remapping is possible.

·         Override the GetValue method to return the value of a column for the current row.

·         Override the Read method to advance to the next row.

·         If the subclass contains disposable resources, override the dispose method.

For example:

privateclassBulkDataReaderSampleData : BulkDataReader
{
    privatestring schemaName;

    privatestring tableName;

    privateint sampleSize;

    privateSampleValue sampleValue;
    public BulkDataReaderSampleData(string      schemaName,
                                    string      tableName,
                                    int         sampleSize,
                                    SampleValue sampleValue)
    {
        this.schemaName = schemaName;
        this.tableName = tableName;
        this.sampleSize = sampleSize;
        this.sampleValue = sampleValue;
    }

    protectedoverridestring SchemaName
    {
        get { returnthis.schemaName; }
    }

    protectedoverridestring TableName
    {
        get { returnthis.tableName; }
    }

    protectedoverridevoid AddSchemaTableRows()
    {
        AddSchemaTableRow("Name",    100,  null, null, true,  false, false, SqlDbType.NVarChar, null, null, null, null, null);
        AddSchemaTableRow("DateUtc", null, null, null, false, false, false, SqlDbType.DateTime, null, null, null, null, null);
        AddSchemaTableRow("Level",   null, null, null, false, false, false, SqlDbType.TinyInt,  null, null, null, null, null);
        AddSchemaTableRow("Payload", 1000, null, null, false, false, true, SqlDbType.VarBinary, null, null, null, null, null);
    }

    publicoverrideobject GetValue(int i)
    {
        returnthis.sampleValue(this.readCount, i);
    }

    privateint readCount = 0;

    publicoverridebool Read()
    {
        return readCount++ < this.sampleSize;
    }
}

For brevity the XML document comments have been removed. The download contains more details.

To use the BulkDataReader class to load a table with data, use a pattern like:

using (BulkDataReaderSampleData bulkReader = newBulkDataReaderSampleData(schemaName,
                                                                          tableName,
                                                                          sampleSize,
                                                                          GetSampleValue))
{
    SqlConnectionStringBuilder connectionString = newSqlConnectionStringBuilder()
    {
        ConnectTimeout = 60, // 1 minute
        InitialCatalog = databaseName,
        IntegratedSecurity = true
    };

    using (SqlConnection connection = newSqlConnection(connectionString.ConnectionString))
    {
        connection.Open();

        using (SqlBulkCopy bulkCopier = newSqlBulkCopy(connection,
                                                        SqlBulkCopyOptions.CheckConstraints |
                                                        SqlBulkCopyOptions.FireTriggers |
                                                        SqlBulkCopyOptions.KeepNulls |
                                                        SqlBulkCopyOptions.TableLock |
                                                        SqlBulkCopyOptions.UseInternalTransaction,
                                                        null))
        {
            foreach (SqlBulkCopyColumnMapping mapping in bulkReader.ColumnMappings)
            {
                bulkCopier.ColumnMappings.Add(mapping);
            }

            using (SqlCommandBuilder quoter = newSqlCommandBuilder())
            {
                bulkCopier.DestinationTableName = quoter.QuoteIdentifier(schemaName) + "." + quoter.QuoteIdentifier(tableName);
                bulkCopier.BulkCopyTimeout = 60; // 1 minute
                bulkCopier.BatchSize = 1000;
                bulkCopier.EnableStreaming = true;

                bulkCopier.WriteToServer(bulkReader);
            }
        }
    }
}

This is similar to the pattern for a DataTable.

Care needs to be taken in the choice of bulk loading options. The choices can noticeably impact overall performance and throughput. The choices in the test code are for a case where multiple data sources are bulk loading data and readers are using snapshot isolation.

Results

To test the performance implications of various combinations of compression, clustering key types and loading techniques I used the test code in the download to load tables like this:

create table [LoadTest].[IntKeyCompressed]
(
  [Id]      int not null
              constraint [Default_IntKeyCompressed_Id] default (next value for [LoadTest].[IntKeySequence])
              constraint [PK_IntKeyCompressed] primary key clustered
                with (fillfactor = 100),
  [Name]    nvarchar(100) not null
              constraint [Check_IntKeyCompressed_Name_NotEmpty] check ([Name] <> N'')
              constraint [Unique_IntKeyCompressed_Name] unique
                 with (fillfactor = 80,
                       data_compression = row),
  [DateUtc] datetime2 not null,
  [Level]   tinyint not null
              constraint [FK_IntKeyCompressed_Levels_Level] references [LoadTest].[Levels]([Id]),
  [Payload] varbinary(1000) null
)
with (data_compression = row);

create index [IX_IntKeyCompressed_Level] on [LoadTest].[IntKeyCompressed]([Level])
  with (fillfactor = 80,
        data_compression = row);

create index [IX_IntKeyCompressed_DateUtc] on [LoadTest].[IntKeyCompressed]([DateUtc])
  with (fillfactor = 100,
        data_compression = row);

The tables approximate event logs. Event logs frequently have high insert volumes. The table also has typical constraints and indexes to correct for the overhead associated with them. MSTest was used for timing the runs and to minimize variability. Since the variability was low, only 3 test runs per test case was necessary. The download can be easily modified for other table structures.

Table 1 shows the time to load 1 000 000 rows into an empty table. The SQL Server 2012 database was running in a virtual machine on a workstation so much higher performance can be expected for realistic hardware. For example, a low end database server was able to load 115 692 SharePoint ULS events per second (1 000 000 rows in 8.6 seconds) using the BulkDataReader. Table 2 shows the time to load 1 000 000 rows of data into a table with 1 000 000 rows. The initial table had been rebuilt to simulate normal database maintenance. The natural key based table has one less index since it does not have a surrogate key thus its better performance than the GUID based clustering key.

Table 1: Time, in seconds, to load 1 000 000 rows of data into an empty table for compressed and uncompressed tables, various clustering key types and various loading techniques. The times are a mean of 3 runs.

Table 2: Time, in seconds, to load 1 000 000 rows of data into a table with 1 000 000 rows for compressed and uncompressed tables, various clustering key types and various loading techniques. The times are a mean of 3 runs.

As expected, bulk loading data is more efficient than single inserts, integer based surrogate keys are more efficient than alternatives and heaps are an efficient way to stage data.

Download

The download is a Visual Studio 2012 solution. It contains:

·         The DataUtilities C^# project contains the code for the BulkDataReader class.

·         The DataUtilitiesTest C^# test project contains the test code for the correctness of the BulkDataReader class and performance tests.

·         The TestDatabase SQL Server database project contains the definition of the test tables.

FastLoader.zip

Introduction

A common cause of mysterious performance issues are traces that have been left running. Naïve use of traces can leave many traces running—traces that are slowing down the application by consuming critical resources. This happens because the SQL Server Performance Analyzer process is killed or trace sessions forgotten. In analyzing an issue, many traces may be created and left running.

Scripts

I see this issue enough that I have created some scripts for the issue. The following query will tell you which non-default traces exist:

use [master]

select T.id as [Id],
       case T.status when 0 then N'stopped' else N'running' end as [Status],
       T.path as [Path],
       case T.is_rowset when 0 then N'false' else N'true' end as [Rowset],
       case T.is_shutdown when 0 then N'disabled' else N'enabled' end as [Shutdown option],
       T.start_time as [Start],
       T.stop_time as [Stop]
from sys.traces as T
where T.is_default <> 1;

The following query can be used to close all non-default traces it also lists their locations.

use [master];

set nocount on;

declare @result table ([Path] nvarchar(260) not null);

declare trace cursor local fast_forward for select T.id, T.path
                                            from sys.traces as T
                                            where T.is_default != 1;
                                            
open trace;

declare @id int;
declare @path nvarchar(260);

fetch next from trace
  into @id, @path;
 
while @@fetch_status = 0
begin
   execute sp_trace_setstatus
     @traceid = @id,
     @status = 0;
    
   execute sp_trace_setstatus
     @traceid = @id,
     @status = 2;
    
   if (@path is not null)
   begin
     insert into @result([Path])
       values (@path);
   end;

       fetch next from trace
         into @id, @path;
end;

close trace;
deallocate trace;

select R.[Path] as [Path]
from @result as R;

In general, the queries require view server state permission:

grant alter trace to [login name];

Introduction

I frequently help teams inside and outside of Microsoft investigate SQL Server database performance issues. As well, I monitor some large internal SharePoint databases to understand their performance characteristics and to make sure we are improving our overall performance. Here are some of the queries I use to investigate performance issues and monitor performance. I hope you will find them useful.

SQL Server provides an extensive list of views and functions that are useful for understanding the internal behavior of SQL Server (see Microsoft SQL Server 2008 Internals by Kalen Delaney et al for a good explanation of SQL Server’s internals). Performance issues manifest themselves as excessive use of some system resource. In most applications, the resources of interest are memory and CPU. These are of interest in SQL Server but also of interest are resources like locks, latches and storage bandwidth.

Most SQL Server resources are managed at the SQL Server instance level. In general, a SQL Server instance should correspond to a single application. Thus the queries below work at the SQL Server level. Where sensible, I have added a commented out line that will scope the query to the current database.

Performance analysis and improvements is a tradeoff. It may not be cost effective to fix minor issues. Thus the queries below focus on highlighting the significant issues.

In general, the queries require view server state permission:

grant view server state to [login name];

Storage operations

SQL Server applications are frequently limited by disk bandwidth. Whenever SQL Server reads a data page that is called a logical read. When the page is not in memory and thus must be read from disk that is called a physical read. The situation is slightly different for writes. Whenever SQL Server changes a data page that is called a logical write. However, the page is not immediately written to disk. Instead it is queued and the lazy writer will eventually write it to disk. Since the page may be modified by several T-SQL statements before it is written to disk, a single T-SQL statement cannot be directly associated with the write. For example, code that appends rows to the end of a table may make many logical writes but cause relatively few physical writes.

Module logical reads

Here is a query to find the modules (i.e. procedures and triggers) with the highest logical read volumes.

select top(10) D.total_physical_reads as [Total physical reads],
               D.total_logical_reads as [Total logical reads],
               D.execution_count as [Executions],
               coalesce(quotename(db_name(D.database_id)), N'-') + N'.' + coalesce(quotename(object_schema_name(D.object_id, D.database_id)), N'-') + N'.' + coalesce(quotename(object_name(D.object_id, D.database_id)), N'-') as [Module],
               QP.query_plan as [Query plan]
from sys.dm_exec_procedure_stats as D
     outer apply sys.dm_exec_query_plan(D.plan_handle) as QP
--where D.database_id = db_id()
order by [Total logical reads] desc;

If a module is performing more logical reads than expected, it may indicate poor logic or missing indexes. Adding the missing indexes will make some updates more expensive but overall it may improve the performance of the application. In SQL Server Management Studio (SSMS), you can click on the result set’s query plan column to see the query plan and further investigate the cause of unexpected logical reads.

Logical reads, that do not result in physical reads, may cause issues if the pages should not otherwise be in memory. Essentially, the module is causing memory to be wasted that might be better used elsewhere or not even be required.

Module physical reads

Here is a query to find the modules with the highest physical read volumes.

select top(10) D.total_physical_reads as [Total physical reads],
               D.total_logical_reads as [Total logical reads],
               D.execution_count as [Executions],
               coalesce(quotename(db_name(D.database_id)), N'-') + N'.' + coalesce(quotename(object_schema_name(D.object_id, D.database_id)), N'-') + N'.' + coalesce(quotename(object_name(D.object_id, D.database_id)), N'-') as [Module],
               QP.query_plan as [Query plan]
from sys.dm_exec_procedure_stats as D
     outer apply sys.dm_exec_query_plan(D.plan_handle) as QP
--where D.database_id = db_id()
order by [Total physical reads] desc;

If a module is performing more physical reads than expected, it may indicate poor logic or missing indexes. Since physical reads take much more time than pure logical reads they are a major cause of latency in database operations.

Statement logical reads

Often individual T-SQL statements cause high logical and physical read volumes. Here is a query to find statements with the highest logical read volumes.

select top(10) sum(Q.total_physical_reads) as [Physical reads],
               sum(Q.total_logical_reads) as [Logical reads],
               sum(Q.execution_count) as [Execution count],
               min(coalesce(quotename(db_name(Q.dbid)), N'-') + N'.' + coalesce(quotename(object_schema_name(Q.objectid, Q.dbid)), N'-') + N'.' + coalesce(quotename(object_name(Q.objectid, Q.dbid)), N'-')) as [Module],
               min(Q.statement_text) as [Statement]
from (select QS.*,
             ST.dbid as dbid,
             ST.objectid as objectid,
             substring(ST.text,
                      (QS.statement_start_offset/2) + 1,
                      ((case statement_end_offset
                        when -1 then datalength(ST.text)
                        else QS.statement_end_offset end
                        - QS.statement_start_offset)/2) + 1) as statement_text
      from sys.dm_exec_query_stats as QS
           cross apply sys.dm_exec_sql_text(QS.sql_handle) as ST) as Q
--where Q.dbid = db_id()
group by Q.query_hash
order by [Logical Reads] desc;

Statement physical reads

Here is a query to find statements with the highest physical read volumes.

select top(10) sum(Q.total_physical_reads) as [Physical reads],
               sum(Q.total_logical_reads) as [Logical reads],
               sum(Q.execution_count) as [Execution count],
               min(coalesce(quotename(db_name(Q.dbid)), N'-') + N'.' + coalesce(quotename(object_schema_name(Q.objectid, Q.dbid)), N'-') + N'.' + coalesce(quotename(object_name(Q.objectid, Q.dbid)), N'-')) as [Module],
               min(Q.statement_text) as [Statement]
from (select QS.*,
             ST.dbid as dbid,
             ST.objectid as objectid,
             substring(ST.text,
                      (QS.statement_start_offset/2) + 1,
                      ((case statement_end_offset
                        when -1 then datalength(ST.text)
                        else QS.statement_end_offset end
                        - QS.statement_start_offset)/2) + 1) as statement_text
      from sys.dm_exec_query_stats as QS
           cross apply sys.dm_exec_sql_text(QS.sql_handle) as ST) as Q
--where Q.dbid = db_id()
group by Q.query_hash
order by [Physical Reads] desc;

These two queries can be particularly useful in environments where the application runs queries directly.

Module writes

Module and statement logical write volumes are often not comparable across modules and statements. This is because of the lazy writer issue mentioned above. Many logical writes that append to an index may have lower actual impact than a few writes to random places in the same index. Thus the results of these queries needs to be interpreted carefully.

Here is a query to find the modules with the highest logical write volumes.

select top(10) D.total_logical_writes as [Total logical writes],
               D.execution_count as [Executions],
               coalesce(quotename(db_name(D.database_id)), N'-') + N'.' + coalesce(quotename(object_schema_name(D.object_id, D.database_id)), N'-') + N'.' + coalesce(quotename(object_name(D.object_id, D.database_id)), N'-') as [Module],
               QP.query_plan as [Query plan]
from sys.dm_exec_procedure_stats as D
     outer apply sys.dm_exec_query_plan(D.plan_handle) as QP
--where D.database_id = db_id()
order by [Total logical writes] desc;

Statement writes

Here is a query to find statements with the highest logical write volumes.

select top(10) sum(Q.total_logical_writes) as [Logical writes],
               sum(Q.execution_count) as [Execution count],
               min(coalesce(quotename(db_name(Q.dbid)), N'-') + N'.' + coalesce(quotename(object_schema_name(Q.objectid, Q.dbid)), N'-') + N'.' + coalesce(quotename(object_name(Q.objectid, Q.dbid)), N'-')) as [Module],
               min(Q.statement_text) as [Statement]
from (select QS.*,
             ST.dbid as dbid,
             ST.objectid as objectid,
             substring(ST.text,
                      (QS.statement_start_offset/2) + 1,
                      ((case statement_end_offset
                        when -1 then datalength(ST.text)
                        else QS.statement_end_offset end
                        - QS.statement_start_offset)/2) + 1) as statement_text
      from sys.dm_exec_query_stats as QS
           cross apply sys.dm_exec_sql_text(QS.sql_handle) as ST) as Q
-- where Q.dbid = db_id()
group by Q.query_hash
order by [Logical Writes] desc;

Indexes

Often adding indexes can improve the performance of critical queries. Here is a query to find the indexes that are most likely to improve the overall performance of the database application if added.

select coalesce(quotename(db_name(MID.database_id)), N'-') + N'.' + coalesce(quotename(object_schema_name(MID.object_id, MID.database_id)), N'-') + N'.' + coalesce(quotename(object_name(MID.object_id, MID.database_id)), N'-') as [Object],
       round(MIGS.avg_total_user_cost *(MIGS.avg_user_impact/100.0) *(MIGS.user_seeks + MIGS.user_scans), 0) as [Relative cost],
       round(MIGS.avg_total_user_cost, 2) [Average cost],
       round(MIGS.avg_user_impact, 2) [Percentage improvement],
       MIGS.user_scans as [User scans],
       MIGS.user_seeks as [User seeks],
       coalesce(MID.equality_columns, N'') as [Equi-join],
       coalesce(MID.inequality_columns, N'') as [Inequi-join],
       coalesce(MID.included_columns, N'') as [Included]
from sys.dm_db_missing_index_group_stats as MIGS
     inner join sys.dm_db_missing_index_groups as MIG on MIG.index_group_handle = MIGS.group_handle
     inner join sys.dm_db_missing_index_details as MID on MID.index_handle = MIG.index_handle
--where MID.database_id = db_id()
order by [Percentage Improvement] desc;

Indexes should be added judiciously since they will slow down some parts of the database application (e.g. updates to the columns in the index) and increase the storage size of the database.

Sometimes database applications contain indexes that are rarely used. These indexes slow down updates and increase storage and related costs (e.g. backup). Here is a query to find the usage of all indexes in a database.

select coalesce(quotename(db_name(US.database_id)), N'-') + N'.' + coalesce(quotename(object_schema_name(US.object_id, US.database_id)), N'-') + N'.' + coalesce(quotename(object_name(US.object_id, US.database_id)), N'-') as [Object],
       I.name as [Index],
       convert(decimal(38,2), PS.[Page count]/128.0) as [Size (MB)],
       (US.user_seeks + US.user_scans + US.user_lookups) as [User operations],
       US.user_seeks as [User seeks],
       US.user_scans as [User scans],
       US.user_lookups as [User bookmark lookups],
       US.user_updates as [User updates]
from sys.dm_db_index_usage_stats as US
     cross apply(select IPS.object_id, IPS.index_id, sum(IPS.page_count)
                  from sys.dm_db_index_physical_stats(US.database_id, US.object_id, US.index_id, null, N'SAMPLED' /* For non-leaf pages use N'DETAILED'*/) as IPS
                  group by IPS.object_id, IPS.index_id) as PS([Object], [Index], [Page count])
     inner join sys.indexes as I on I.object_id = US.object_id and
                                    I.index_id = US.index_id
where D.database_id = db_id()
order by [User Operations], [Object], [Index];

Indexes with low use are candidates for removal.

Indexes may become fragmented in these ways:

1.       The pages may become logically out-of-order on disk. This was once undesirable because logical order was closely related to physical order, sequential disk access was faster than non-sequential access and indexes are frequently scanned in page order. However, this is rarely an issue now because RAID, SANs, Windows Server storage and many disk drives and controllers reorder pages on disk so the logical order does not reflect the physical order and thus pages being in logical order does not give any performance gain. Also, for some technologies like SSDs sequential physical access may give no gain in performance.

2.       Secondly, the pages may be non-full. Often, many pages in an index must be read to complete a SQL Server operation and non-full pages have less data per page than full pages thus non-full pages will often mean more pages need to be read from disk and memory buffers are less efficient. However, when data is inserted into a full page it will split into two half full pages which are less efficient. Thus in general, the best performance is usually gained by a target of pages that are almost full but with enough room for inserts to rarely split a page.

There are numerous considerations in determining if index fragmentation is likely to cause performance issues. For example, SQL Server fetches entire extents (a group of 8 pages of 8Kb each) not pages thus fragmentation in indexes of just a few extents is unlikely to have a performance impact. These considerations lead to the following heuristic based query for determining which indexes need defragmentation in a database.

select coalesce(quotename(D.name), N'-') + N'.' + coalesce(quotename(object_schema_name(IPS.object_id, D.database_id)), N'-') + N'.' + coalesce(quotename(object_name(IPS.object_id, D.database_id)), N'-') as [Object],
       I.name as [Index],
       IPS.partition_number as [Partition],
       IPS.index_level as [Level],
       convert(decimal(38,2), IPS.page_count/128.0) as [Size (MB)],
       IPS.avg_fragmentation_in_percent as [Mean fragementation (%)],
       IPS.avg_page_space_used_in_percent as [Mean  space Used (%)],
       case
         when IPS.avg_fragmentation_in_percent < 30.0 then N'Reorganize'
         else N'Rebuild'
       end as [Action]
from sys.databases as D
     cross apply sys.dm_db_index_physical_stats(D.database_id, null, null, null, N'SAMPLED' /* For non-leaf pages use N'DETAILED'*/) as IPS
     inner join sys.indexes as I on I.object_id = IPS.object_id and
                                    I.index_id = IPS.index_id
where D.database_id = db_id() and
      IPS.alloc_unit_type_desc = 'IN_ROW_DATA' and
      ((IPS.page_count > 24 and IPS.avg_fragmentation_in_percent > 5.0) or             -- Fragmented (only relevant where in-order pages are faster to read or write).
      (IPS.page_count > 8 and IPS.avg_page_space_used_in_percent < I.fill_factor*0.9)) -- Compaction needed.
order by [Object], [Index], [Partition], [Level];

Sometimes, it is more efficient to reorganize an index than to rebuild it. The above query also heuristically determines if the index should be reorganized or rebuilt. Here is a query that does not use any heuristics to filter the results:

select coalesce(quotename(D.name), N'-') + N'.' + coalesce(quotename(object_schema_name(IPS.object_id, D.database_id)), N'-') + N'.' + coalesce(quotename(object_name(IPS.object_id, D.database_id)), N'-') as [Object],
       I.name as [Index],
       IPS.partition_number as [Partition],
       IPS.index_level as [Level],
       convert(decimal(38,2), IPS.page_count/128.0) as [Size (MB)],
       IPS.avg_fragmentation_in_percent as [Mean fragementation (%)], -- Only relevant where in-order pages are faster to read or write.
       IPS.avg_page_space_used_in_percent as [Mean  space Used (%)]
from sys.databases as D
     cross apply sys.dm_db_index_physical_stats(D.database_id, null, null, null, N'SAMPLED' /* For non-leaf pages use N'DETAILED'*/) as IPS
     inner join sys.indexes as I on I.object_id = IPS.object_id and I.index_id = IPS.index_id
where D.database_id = db_id()
order by [Object], [Index], [Partition], [Level];

For efficiency reasons, both queries sample leaf pages in the B+-trees. Using the DETAILED option instead will give data for all pages.

Storage

Storage is usually cheap compared to the engineering cost of reducing it. However, for databases that consume a large amount of storage it may be worth the engineering investment to reduce storage costs. In a storage cost analysis there are usually two values of interest: the number of rows and the mean size of each row. If the number of rows is unexpectedly large there may an error in the application. Often when troubleshooting database size growth this value indicates where the issue is. If the mean row size is unexpectedly large it may indicate over indexing of large values.

Here is a query to find the approximate storage used by database objects

select top(10) db_name() + N'.' + coalesce(quotename(object_schema_name(PS.object_id)), N'-') + N'.' + coalesce(quotename(object_name(PS.object_id)), N'-') as [Object],
               convert(decimal(38,2), sum(PS.used_page_count)/128.0) as [Size (MB)],
               sum(case when PS.index_id in(0, 1) then PS.row_count else 0 end) as [Rows],
               (sum(PS.used_page_count)*8000)/sum(case when PS.index_id in(0, 1) then PS.row_count else 0 end) as [Mean row size (Bytes)]

from sys.dm_db_partition_stats as PS
group by db_name() + N'.' + coalesce(quotename(object_schema_name(PS.object_id)), N'-') + N'.' + coalesce(quotename(object_name(PS.object_id)), N'-')
order by [Size (MB)] desc

The query runs quickly but is based on cached information that may be out-of-date. Here is a query to find the exact storage used by database objects:

select top (10) coalesce(quotename(db_name(D.database_id)), N'-') + N'.' + coalesce(quotename(object_schema_name(PS.object_id, D.database_id)), N'-') + N'.' + coalesce(quotename(object_name(PS.object_id, D.database_id)), N'-') as [Object],
                convert(decimal(38,2), sum(PS.page_count)/128.0) as [Size (MB)]
from sys.databases as D
     cross apply sys.dm_db_index_physical_stats(D.database_id, null, null, null, N'DETAILED') as PS
--where D.database_id = db_id()
group by coalesce(quotename(db_name(D.database_id)), N'-') + N'.' + coalesce(quotename(object_schema_name(PS.object_id, D.database_id)), N'-') + N'.' + coalesce(quotename(object_name(PS.object_id, D.database_id)), N'-')             
order by [Size (MB)] desc

Processor

Most SQL Server application performance issues are related to storage bandwidth. However, occasionally computation bandwidth becomes an issue.

Module processor usage

Here is a query to find the modules that are the most computationally expensive.

select top(10) convert(decimal(38, 6), D.total_worker_time)/1000000 as [CPU (secs)],
               D.execution_count as [Executions],
               coalesce(quotename(db_name(D.database_id)), N'-') + N'.' + coalesce(quotename(object_schema_name(D.object_id, D.database_id)), N'-') + N'.' + coalesce(quotename(object_name(D.object_id, D.database_id)), N'-') as [Module],
               QP.query_plan as [Query plan]
from sys.dm_exec_procedure_stats as D
     outer apply sys.dm_exec_query_plan(D.plan_handle) as QP
--where D.database_id = db_id()
order by D.total_worker_time desc;

Statement usage

Here is a query to find the statements that are the most computationally expensive.

select top(10) sum(convert(decimal(38, 6), Q.total_worker_time)/1000000) as [CPU (secs)],
               sum(Q.execution_count) as [Execution count],
               min(coalesce(quotename(db_name(Q.dbid)), N'-') + N'.' + coalesce(quotename(object_schema_name(Q.objectid, Q.dbid)), N'-') + N'.' + coalesce(quotename(object_name(Q.objectid, Q.dbid)), N'-')) as [Module],
               min(Q.statement_text) as [Statement]
from (select QS.*,
             ST.dbid as dbid,
             ST.objectid as objectid,
             substring(ST.text,
                      (QS.statement_start_offset/2) + 1,
                      ((case statement_end_offset
                        when -1 then datalength(ST.text)
                        else QS.statement_end_offset end
                        - QS.statement_start_offset)/2) + 1) as statement_text
      from sys.dm_exec_query_stats as QS
           cross apply sys.dm_exec_sql_text(QS.sql_handle) as ST) as Q
--where Q.dbid = db_id()
group by Q.query_hash
order by [CPU (secs)] desc;

Waits

Some SQL Server resources cannot be used concurrently. SQL Server uses various mechanisms to control usage of these resources. When one SQL Server process needs to use a resource that another is currently using it must wait for the resource to be freed. Thus one operation can cause another operation to take longer than it would otherwise take.

Wait analysis requires some sophistication with the internals of SQL Server but is vital to performance tuning a high performance database application. To understand the resources in contention, during a time period, run the following statements at the beginning of the time period.

checkpoint;
dbcc freeproccache with no_infomsgs;
dbcc dropcleanbuffers with no_infomsgs;
dbcc sqlperf('sys.dm_os_wait_stats', clear);
dbcc sqlperf('sys.dm_os_latch_stats', clear);

These statements reset internal counters and the state of the server to give more accurate results. At the end of the time period run the following query

select N'Lock total: ' + WS.wait_type as [Type],
       WS.waiting_tasks_count as [Waits (number)],
       convert(decimal(38,2), WS.wait_time_ms/1000.0) as [Total wait time (secs)],
       WS.wait_time_ms/WS.waiting_tasks_count as [Average wait time (ms)]
from sys.dm_os_wait_stats as WS
where WS.waiting_tasks_count > 0
union all
select N'Lock post signal: ' + WS.wait_type,
       WS.waiting_tasks_count,
       convert(decimal(38,2), WS.signal_wait_time_ms/1000.0),
       WS.signal_wait_time_ms/WS.waiting_tasks_count
from sys.dm_os_wait_stats as WS
where WS.waiting_tasks_count > 0
union all
select N'Latch total: ' + LS.latch_class,
       LS.waiting_requests_count,
       convert(decimal(38,2), LS.wait_time_ms/1000.0),
       LS.wait_time_ms/LS.waiting_requests_count
from sys.dm_os_latch_stats as LS
where LS.waiting_requests_count > 0
order by [Total wait time (secs)] desc;

See https://msdn.microsoft.com/en-us/library/ms179984.aspx for a description of wait types. To run these two scripts you will need to be a member of either the sysadmin or serveradmin roles.

For a deep analysis, use the Windows Performance Toolkit to understand the deep behavior of a SQL Server application. SQL Server publishes data about its internals through ETW and this data can be combined with other ETW traces to build a detailed understanding of resource contention issues.

Wait analysis can be useful in troubleshooting the cause of latency in a database application. The following script reports which statements are blocking other statements (i.e. using a resource the blocked statement is waiting for).

set nocount on;

while 1=1
begin
    declare @results table (
                               [Id]                       int identity not null primary key,
                               [Blocked module]           nvarchar(776) not null,
                               [Blocked statement]        nvarchar(max) not null,
                               [Blocking module]          nvarchar(776) null,
                               [Blocking statement]       nvarchar(max) null,
                               [Wait type]                nvarchar(60) not null,
                               [Total wait time (MS)]     bigint not null,
                               [Sessions waiting (count)] int not null
                           );
                          
    delete from @results;
   
    with Statements([Blocked module], [Blocked statement start], [Blocked statement end],
                     [Blocking module], [Blocking statement start], [Blocking statement end],
                     [Wait type], [Total wait time (MS)], [Sessions waiting (count)])
    as
    (
      select BlockedR.sql_handle as [Blocked Module],
             BlockedR.statement_start_offset as [Blocked Statement start],
             BlockedR.statement_end_offset as [Blocked Statement end],
             BlockingR.sql_handle as [Blocking Module],
             BlockingR.statement_start_offset as [Blocking statement start],
             BlockingR.statement_end_offset as [Blocking statement end],
             WT.wait_type as [Wait type],
             sum(convert(bigint, WT.wait_duration_ms)) as [Total wait time (MS)],
             count(*) as [Sessions waiting (count)]
      from sys.dm_os_waiting_tasks as WT
           inner join sys.dm_exec_requests as BlockedR on BlockedR.session_id = WT.session_id
           left outer join sys.dm_exec_requests as BlockingR on BlockingR.session_id = WT.blocking_session_id
      group by BlockedR.sql_handle, BlockedR.statement_start_offset, BlockedR.statement_end_offset,
               BlockingR.sql_handle, BlockingR.statement_start_offset, BlockingR.statement_end_offset,
               WT.wait_type
    )
    insert into @results([Blocked module], [Blocked statement],
                          [Blocking module], [Blocking statement],
                          [Wait type], [Total wait time (MS)], [Sessions waiting (count)])
    select coalesce(quotename(db_name(BlockedText.dbid)), N'-') + N'.' + coalesce(quotename(object_schema_name(BlockedText.objectid, BlockedText.dbid)), N'-') + N'.' + coalesce(quotename(object_name(BlockedText.objectid, BlockedText.dbid)), N'-') as [Blocked module],
           substring(BlockedText.text,
                     (S.[Blocked statement start]/2) + 1,
                     ((case S.[Blocked statement end]
                         when -1 then datalength(BlockingText.text)
                         else S.[Blocked statement end]
                       end
                       - S.[Blocked statement start])/2) + 1) as [Blocked statement],
           coalesce(quotename(db_name(BlockingText.dbid)), N'-') + N'.' + coalesce(quotename(object_schema_name(BlockingText.objectid, BlockingText.dbid)), N'-') + N'.' + coalesce(quotename(object_name(BlockingText.objectid, BlockingText.dbid)), N'-') as [Blocking module],
           substring(BlockingText.text,
                     (S.[Blocking statement start]/2) + 1,
                     ((case S.[Blocking statement end]
                         when -1 then datalength(BlockingText.text)
                         else S.[Blocking statement end]
                       end
                       - S.[Blocking statement start])/2) + 1) as [Blocking statement],
           S.[Wait type] as [Wait type],
           S.[Total wait time (MS)] as [Total wait time (MS)],
           S.[Sessions waiting (count)] as [Sessions waiting (count)]
    from Statements as S
         cross apply sys.dm_exec_sql_text(S.[Blocked module]) as BlockedText
         cross apply sys.dm_exec_sql_text(S.[Blocking module]) as BlockingText
    --where S.[Total wait time (MS)] > 50
    order by S.[Total wait time (MS)] desc
    option (force order);

    if exists(select * from @results)
    begin
      select getdate(),
             [Blocked module], [Blocked statement],
           [Blocking module], [Blocking statement],
           [Wait type], [Total wait time (MS)], [Sessions waiting (count)]
      from @results;
    end;

    waitfor delay '00:00:01'
end

Every second the script reports the blocked and blocking statements. Care is needed in interpreting the report because a blocked statement may itself be blocked waiting for another resource.

Script

The downloadable T-SQL script contains the above queries and some less commonly used scripts for:

·         Finding the databases that have the highest read volumes.

·         Finding the databases that have the highest write volumes.

·         Finding the total SQL Server instance read and write volume.

·         The modules that are recompiled the most.

Performance investigation.sql

Calculating the size of an index

If there are variable length columns, SQL Server may allow the uniqueness constraint to be created but oversized rows will not be insertable. Below is a formula to determine when an index may overflow the 900 byte limit. The maximum size of a unique constraint index key can be calculated using Table 1 as:

 where  is the overhead,  is the variable sized column overhead for  variable sized columns,  is the nullable column overhead for  nullable columns,  is the type cost for column ,  is the number of variable sized columns, is the number of nullable columns and  is the number of columns. The overhead  varies depending on whether or not the underlying index is clustered (see Table 1).

Table 1 Table of contributions to the size of an index key. To calculate the maximum size of an index key in bytes add the size of every column in the index key and the primary key and add the overhead.

Using hash indexes to enforce uniqueness constraints

If the index key will always be 900 bytes or less, then an ordinary uniqueness constraint may be used. If the index key could be greater than 900 bytes, then a uniqueness constraint can be enforced with an after-trigger. To avoid table scans, a hash index can be generated for the columns with the unique constraint. The uniqueness constraint can be enforced with the scheme:

create table [Schema].[Table]
(
  [Id]      int not null constraint [PK_Table] primary key,
  …,
  [Column₁] Column₁TypeDeclaration not null,
  [Column₂] Column₂TypeDeclaration null,
  …,
  [Internal_Hash_Column₁_Column₂] as checksum([Column₁], [Column₂]) persisted not null
);
go

create index [Unique_Hash_Table_Column₁_Column₂] on [Schema].[Table] ([Internal_Hash_Column₁_Column₂]);
go

create trigger [ConstraintInsertTrigger_Extent] on [Schema].[Table]
  after of insert, update as
begin
  if (update([Column₁]) or update([Column₂])) and
      exists (select *
              from [Schema].[Table] as T
                   inner join inserted as I on I.[Internal_Hash_Column₁_Column₂] = T. [Internal_Hash_Column₁_Column₂] and
                                               I.[Column₁] = T.[Column₁] and
                                               (I.[Column₂] = T.[Column₂] or (I.[Column₂] is null and T.[Column₂] is null))
       group by I.[Internal_Hash_Column₁_Column₂], I.[Column₁], I.[Column₂]
       having count(*) > 1)
  begin
    if @@trancount > 0
    begin
      rollback transaction;
    end;

    raiserror(N'Error: Values in the columns [Schema].[Table] ([Column₁], [Column₂]) must be unique.', 16, 0);
  end;
end;
go

Essentially, the strategy is to store an indexed hash of the columns in the uniqueness constraint. The after trigger makes sure there are no duplicates, by searching for rows that both hash to the same value as one of the inserted rows and match on the constrained columns. Using hashes avoids scanning the entire table for matches and has good space efficiency. The above scheme shows how to handle multiple columns and both nullable and non-nullable columns.

The reason to use checksum and neither hashbytes nor binary_checksum is that checksum respects collation. For example,

checksum(N'X') = checksum(N'x')

but

binary_checksum(N'X') ≠ binary_checksum(N'x').

However, hashbytes and binary_checksum are suitable hashing functions for non-text columns. Also if the constrained columns are unlikely to contain common prefixes, a prefix of the columns (e.g. left([Column], 10)) may be a very good hashing function.

Using a uniqueness constraint on a hashed column to enforce uniqueness constraints

A common alternative strategy is to put a unique constraint over a persisted column defined using hashbytes. The idea is that the chance of a collision for, say, a SHA1 hash is so low that other bugs or hardware failures are much more likely to cause a failure. However, because of the Birthday Fallacy/Paradox people often underestimate the chance of a collision happening. If a hashing function is perfect then the chance of there being a collision among n randomly chosen values in a b-bit hash is approximately:

 In general, a collision is highly likely in a sample of size

Thus for, say, the 160 bit SHA1 hash the chance of a collision in even a trillion random rows seems very low. However, there are some flaws in this argument:

·         Cryptographic hashing functions are poorly understood but there are strong indications that MD5 and SHA1 are weaker than perfect hashing functions.

·         For a given value, it is often feasible to find another value that hashes to the same hash (i.e. a hash collision). Thus if an application could be compromised by a hash collision and an adversary has some control over the values the application uses then this strategy is unsuitable.

·         Cryptographic hash functions do not respect collations. Thus, two strings that are equal like ‘X’ and ‘x’ (in a case insensitive collation) may hash to different hashes.

·         The input to the hashing function is usually not random.

Relative performance of the hash index approach for columns under 900 bytes

It seems likely that the hash index approach is more efficient than the unique index based approach at enforcing uniqueness constraints. Table 2 and Table 3 show the average time to insert and delete 1 000 and 100 000 rows for various sized strings with a uniqueness constraint. The SQL Server unique constraint and the hash index approach are compared for both the no prefix and the 20 byte (10 Unicode character) common prefix cases. From this analysis, it is clear that for column sets under 900 bytes the hash index approach is unlikely to perform better than a traditional unique index based approach. The test was carried out on a dedicated workstation with an Intel Pentium D 3.00GHz processor and 4 GB of RAM. The test script is attached to this blog entry.

Table 2: The average time, in seconds, to insert and delete 1 000 rows with a Unicode string with a uniqueness constraint. Both the unique and hash index approaches to uniqueness constraint enforcement are tabulated. Both random strings and strings with a fixed 20 byte (10 Unicode character) prefix are tabulated. The first time in each pair, is the time to insert 1 000 rows at once. The second time in each pair, is the time to insert 1 000 rows separately (i.e. 1 at a time).

Table 3: The average time, in seconds, to insert and delete 100 000 rows with a Unicode string with a uniqueness constraint. Both the unique and hash index approaches to uniqueness constraint enforcement are tabulated. Both random strings and strings with a fixed 20 byte (10 Unicode character) prefix are tabulated. The first time in each pair, is the time to insert 100 000 rows at once. The second time in each pair, is the time to insert 100 000 rows separately (i.e. 1 at a time).

Large uniqueness constraints.docx

In Microsoft, like most organizations, it is hard to get things done when you are in different organizational units and on different schedules. This move makes the repository team firmly part of the SQL Server team and on the same schedule. Thus this change will make alignment and deep integration of the repository with rest of the SQL Server platform easier. Also it provides a simple setup experience and much simpler licensing model for customers. Already, there are many planned changes for the next version of SQL Server that will make for a much stronger metadata store and overall platform.

The most commonly advocated SQL Server paging approach is based on the row_number ranking function.

with NumberedResult as
(
  select row_number() over (order by M.[Id]) as [Row], M.[Id], M.[Name]
  from [System_Runtime].[Methods] as M
)
select NR.[Id], NR.[Name]
from NumberedResult as NR
where NR.[Row] between @startRow and @stopRow;

This works by associating a row number with each row and fetching blocks of row numbers. For this to work correctly there must be a fixed row order defined by the over expression and the result set must be unchanging. Otherwise each call to the query may return results in a different order or even different results destroying the illusion of paging through a point in time result set.

A generally better approach is to use top to retrieve a page of data.

select top(@pageSize) M.[Id], M.[Name]
from [System_Runtime].[Methods] as M
where M.[Id] > @lastId
order by M.[Id]

As before, for this to work correctly there must be a fixed row order defined by the order by clause and the result set must be unchanging. The reason this will usually work better is that if the order by clause is covered by an index then the data can be found without sorting the entire data set. In theory, row_number could work this way also but in practice does not. The difference can be seen by looking at query plans—typically a row_number based query will form the entire result set while a top based approach will seek to the first index entry and then return the next page of rows in the index.

If there is no fixed row order or the result set can change then data can be read into a temporary table and processed by either of the two previous approaches.

ADO.Net has good support for paging data https://msdn.microsoft.com/en-us/library/tx1c9c2f.aspx. The Fill method has a specific overload to support paging in of data.

Below I compare the approaches for two interesting performance questions for a table of .Net Framework with 207 966 rows:

·         What is the cost of retrieving the first page?

·         What is the cost of retrieving all the pages?

Below are the results in seconds to perform the tests 100 times. As always it is wise to not make assumptions in performance or concurrency critical code and to test various approaches yourself.

When applicable, the top approach has clear performance advantages. Also it will typically use less server resources than the other approaches.

Here are the test scripts (run with the query option to discard the result set after execution but to collect client statistics).

-- row_number approach: First page
dbcc freesystemcache ('All')

declare @repetitions int = 100;

declare @i int = 0;
while @i < @repetitions
begin
  declare @pageSize bigint = 100;
  declare @lastrow bigint = (select count(*) from [System_Runtime].[Methods]);

  declare @startRow bigint = 1;

  declare @stopRow bigint = @startRow + @pageSize - 1;

  with NumberedResult as
  (
    select row_number() over (order by M.[Id]) as [Row], M.[Id], M.[Name]
    from [System_Runtime].[Methods] as M
  )
  select NR.[Id], NR.[Name]
  from NumberedResult as NR
  where NR.[Row] between @startRow and @stopRow;

  set @i += 1;
end;

-- row_number approach: All pages
dbcc freesystemcache ('All')

declare @repetitions int = 100;

declare @i int = 0;
while @i < @repetitions
begin
  declare @pageSize bigint = 100;
  declare @lastrow bigint = (select count(*) from [System_Runtime].[Methods]);

  declare @startRow bigint = 1;

  while @startRow < @lastRow
  begin
    declare @stopRow bigint = @startRow + @pageSize - 1;

    with NumberedResult as
    (
      select row_number() over (order by M.[Id]) as [Row], M.[Id], M.[Name]
      from [System_Runtime].[Methods] as M
    )
    select NR.[Id], NR.[Name]
    from NumberedResult as NR
    where NR.[Row] between @startRow and @stopRow;

    set @startRow += @pageSize;
  end;

  set @i += 1;
end;

-- top approach: First page
dbcc freesystemcache ('All')

declare @repetitions int = 100;

declare @i int = 0;
while @i < @repetitions
begin
  declare @pageSize bigint = 100;
  declare @maxId bigint = (select max(M.Id) from [System_Runtime].[Methods] as M);

  declare @lastId bigint = 0;

  select top(@pageSize) M.[Id], M.[Name]
  from [System_Runtime].[Methods] as M
  where M.[Id] > @lastId
  order by M.[Id]

  set @i += 1;
end;

-- top approach: All pages
dbcc freesystemcache ('All')

declare @repetitions int = 100;

declare @i int = 0;
while @i < @repetitions
begin
  declare @pageSize bigint = 100;
  declare @maxId bigint = (select max(M.Id) from [System_Runtime].[Methods] as M);

  declare @lastId bigint = 0;

  while @lastId < @maxId
  begin
    select top(@pageSize) M.[Id], M.[Name]
    from [System_Runtime].[Methods] as M
    where M.[Id] > @lastId
    order by M.[Id]

    set @lastId += @pageSize;
  end;

  set @i += 1;
end;

-- temporary table approach: First page
dbcc freesystemcache ('All')

declare @repetitions int = 100;

declare @i int = 0;
while @i < @repetitions
begin
  declare @pageSize bigint = 100;

  create table #results
  (
    [Row]  bigint not null primary key,
    [Id]   bigint not null,
    [Name] nvarchar(max) not null
  );

  insert into #results ([Row], [Id], [Name])
    select row_number() over (order by M.[Id]) as [Row], M.[Id], M.[Name]
    from [System_Runtime].[Methods] as M

  declare @startRow bigint = 1;

  declare @stopRow bigint = @startRow + @pageSize - 1;

  select R.[Id], R.[Name]
  from #results as R
  where R.[Row] between @startRow and @stopRow;

  drop table #results;

  set @i += 1;
end;

-- temporary table approach: All pages
dbcc freesystemcache ('All')

declare @repetitions int = 100;

declare @i int = 0;
while @i < @repetitions
begin
  declare @pageSize bigint = 100;

  create table #results
  (
    [Row]  bigint not null primary key,
    [Id]   bigint not null,
    [Name] nvarchar(max) not null
  );

  insert into #results ([Row], [Id], [Name])
    select row_number() over (order by M.[Id]) as [Row], M.[Id], M.[Name]
    from [System_Runtime].[Methods] as M

  declare @lastrow bigint = (select count(*) from #results);
  declare @startRow bigint = 1;

  while @startRow < @lastRow
  begin
    declare @stopRow bigint = @startRow + @pageSize - 1;

    select R.[Id], R.[Name]
    from #results as R
    where R.[Row] between @startRow and @stopRow;

    set @startRow += @pageSize;
  end;

  drop table #results;

  set @i += 1;
end;

Application lifecycle management (ALM): Facilitating team development of software in a continuous cycle of steps (that is, definition, design, development, testing, deployment, and management). Traditionally, Source Code Control Systems (SCCS) and build systems, like Visual Studio Team Foundation Server, automate ALM.

Schema evolution: Changing schemas over time in such a way that applications continue to run unchanged or with minimal changes.

"Oslo" product versioning: Upgrading a customer's "Oslo" repository, from version n to version n+1, in such a way that the data is preserved.

Application versioning: The ability to upgrade applications, stored in the "Oslo" repository, to new versions. The upgrade process has the option to retain the data from older versions of the application.

Operational versioning: The ability to understand operational/management changes to a deployed application.