ALTER DATABASE HealthDB SET PARAMETERIZATION FORCED

    SELECT * FROM CurrentHealthReports
    WHERE ProjectToCluster = 1

    |--Parallelism(Gather Streams)
        |--Clustered Index Scan(OBJECT:([CurrentHealthReports].[PKCurrentHealthReports]), 
            WHERE:([CurrentHealthReports].[ProjectToCluster]=[@0]))

    |--Nested Loops(Inner Join, OUTER REFERENCES:([CurrentHealthReports].[PKCol]))
        |--Index Seek(OBJECT:([CurrentHealthReports].[idx_NotYetProjected]), 
            SEEK:([CurrentHealthReports].[ProjectToCluster]=(1)) ORDERED FORWARD)
        |--Clustered Index Seek(OBJECT:([CurrentHealthReports].[PKCurrentHealthReports]), 
            SEEK:([CurrentHealthReports].[PKCol]=[CurrentHealthReports].[PKCol]) LOOKUP ORDERED FORWARD)

    SELECT * FROM CurrentHealthReports WITH (INDEX = idx_NotYetProjected)
    WHERE ProjectToCluster = 1

    Msg 8622, Level 16, State 1, Line 48
    Query processor could not produce a query plan because of the hints defined in this query.
    Resubmit the query without specifying any hints and without using SET FORCEPLAN.

    CREATE INDEX idx_NotYetProjected ON CurrentHealthReports (ProjectToCluster)
    WHERE (ProjectToCluster = 1)

    select * from CurrentHealthReports where ProjectToCluster = @0

    ALTER DATABASE HealthDB SET PARAMETERIZATION SIMPLE

    DECLARE @stmt nvarchar(max);
    DECLARE @params nvarchar(max);
    EXEC sp_get_query_template 
        N'SELECT * FROM CurrentHealthReports WHERE ProjectToCluster = 1',
        @stmt OUTPUT,
        @params OUTPUT;
    EXEC sp_create_plan_guide
        N'PlanGuide_HealthNotYetProjectedQuery',
        @stmt,
        N'TEMPLATE',
        NULL,
        @params,
        N'OPTION(PARAMETERIZATION SIMPLE)'

    SELECT * FROM CurrentHealthReports
    WHERE ProjectToCluster = 1
    OPTION (RECOMPILE)

    USE tempdb
    GO
    IF OBJECT_ID ('even') IS NOT NULL DROP TABLE even;
    IF OBJECT_ID ('odd') IS NOT NULL DROP TABLE odd;
    GO
    CREATE TABLE even (c1 int, c2 CHAR(30));
    CREATE TABLE odd (c1 int, c2 CHAR(30));
    GO
    SET NOCOUNT ON;
    DECLARE @x int;
    SET @x = 1;
    BEGIN TRAN;
    WHILE (@x <= 10000)
    BEGIN
        INSERT INTO even (c1, c2) VALUES (@x * 2, '');
        INSERT INTO odd (c1, c2) VALUES (@x * 2 - 1, '');
        IF @x % 1000 = 0
        BEGIN
            RAISERROR ('Inserted %d rows...', 0, 1, @x) WITH NOWAIT;
            COMMIT TRAN;
            BEGIN TRAN;
        END;
        SET @x = @x + 1;
    END;
    WHILE @@TRANCOUNT > 0 COMMIT TRAN;
    GO

    -- QUERY #1
    SELECT *
    FROM even t1
    INNER JOIN even t2 ON t1.c1 = t2.c1

StmtText
------------------------------------------------------------------------------------------
  |--Hash Match(Inner Join, HASH:([t1].[c1])=([t2].[c1]), RESIDUAL:([t2].[c1]=[t1].[c1]))
      |--Table Scan(OBJECT:([tempdb].[dbo].[even] AS [t1]))
      |--Table Scan(OBJECT:([tempdb].[dbo].[even] AS [t2]))

    -- QUERY #2
    SELECT TOP 1 *
    FROM even t1
    INNER JOIN even t2 ON t1.c1 = t2.c1

   Rows  EstRows  Executes  EstExecs StmtText
   ----- -------- --------- --------- -------------------------------------------------------------
   1     1        1         1         |--Top(TOP EXPRESSION:((1)))
   1     1        1         1             |--Nested Loops(Inner Join, WHERE:([t2].[c1]=[t1].[c1]))
   1     1        1         1                 |--Table Scan(OBJECT:([tempdb].[dbo].[even] AS [t1]))
   1     10000    1         1                 |--Table Scan(OBJECT:([tempdb].[dbo].[even] AS [t2]))

     -- QUERY #3
     SELECT TOP 1 *
     FROM even t1
     INNER JOIN odd t2 ON t1.c1 = t2.c1

   Rows      EstRows  Executes  EstExecs StmtText
   --------- -------- --------- --------- -------------------------------------------------------------
   0         1        1         1         |--Top(TOP EXPRESSION:((1)))
   0         1        1         1             |--Nested Loops(Inner Join, WHERE:([t2].[c1]=[t1].[c1]))
   10000     1        1         1                 |--Table Scan(OBJECT:([tempdb].[dbo].[even] AS [t1]))
   100000000 10000    1         2                 |--Table Scan(OBJECT:([tempdb].[dbo].[odd] AS [t2]))

• The “TOP 1” told the query optimizer that it should choose a plan that is likely to return a single row as fast as possible, rather than optimizing for return of all rows that could satisfy the join.
• The data in the [even] and [odd] tables exhibit negative correlation. That is, the values in [even] are never going to match a value in [odd], even though the data distribution is similar. (If you can make a prediction about the data in one column based on the data in another column, you can say that those columns are correlated.)
• There are a few exceptions, but in general you can say that the query optimizer is ignorant of data correlation. In this case, the implication is that the QO can’t know at compile time that it will never find a value in [even] that matches a value in [odd].
• Where the data distributions described by SQL’s statistics indicate overlapping data ranges, the optimizer is generally optimistic, and assumes that it will find a match if it joins those two sets.

    SELECT * FROM Employee
    INNER JOIN Manager ON Employee.ManagerId = Manager.Id

    SELECT * FROM ImportedData
    WHERE RegionCode IN (SELECT RegionCode FROM DisallowedRegions)

    IF EXISTS (
        SELECT * FROM ImportedData
        WHERE RegionCode IN (SELECT RegionCode FROM DisallowedRegions))
    BEGIN
        RAISERROR ('Found some bad data', 16, 1)
    END;

    WHERE (@value = 0) OR ((col1 / @value) = 2)

But the idea is that this variation is functionally-equivalent, and should protect you from the error:

    WHERE
        CASE
            WHEN (@value = 0) THEN 2
            ELSE (col1 / @value)
        END = 2

Before now that’s the advice I would have offered, too. But I just ran into a situation where a CASE statement does not provide predictable short circuiting. Here’s a simplified repro:

    ALTER FUNCTION dbo.test_case_short_circuit (@input INT)
    RETURNS TABLE
    AS
    RETURN (
        SELECT calculated_value =
            CASE
                WHEN @input <= 0 THEN 0
                ELSE LOG10 (@input)
            END
    );
    GO

    SELECT * FROM dbo.test_case_short_circuit (0);
    GO

This fails with this error:


     Msg 3623, Level 16, State 1, Line 2
     An invalid floating point operation occurred.


The LOG10 function raises this error when its input is 0 or a negative value. In some cases it appears that the plan may still evaluate the expression in the second CASE branch even when it won’t be using the value. This is a case where CASE doesn’t provide deterministic short circuiting.

I want to make sure no one takes away the conclusion that SQL Server doesn’t support expression short circuiting. It definitely does. It’s just that you don’t have explicit control over the order of expression evaluation — even with CASE, apparently. And if you’re going to depend on short circuiting, you need a deterministic order of expression evaluation.

What can you do about it? One option would be to always scrub things so that an error isn’t possible even when the CASE branch’s output won’t be used. For example, using “ELSE LOG10 (CASE WHEN @input <= 0 THEN 1 ELSE @input END)” in the repro script doesn’t change the behavior of the function, but avoids the error. Unfortunately, that’s not so pretty.

UPDATE (4 Mar 2011): To be clear, I’ve used CASE before for its short-circuiting properties, and I don’t intend to go back and revisit all of that old code in light of this example. This seems like an edge case to me. But it’s worth being aware that such edge cases exist if you’re thinking about relying on CASE for short circuiting. The most defensive programming approach would be to write the expression in such a way that it doesn’t require particular short circuiting behavior.

UPDATE (10 Jun 2011): The owners of this code have marked this bug as fixed. From their comments, it sounds like you are supposed to be able to rely on deterministic order of expression evaluation for CASE statements. But any SQL release in your hands right now will still be vulnerable to this problem -- keep an eye out for the issue as you use CASE for short circuiting.

(Cross-posted to here.)

Density
Density, when used to describe the data in a column, is a measure of how often duplicate values occur in that column. Another way to think of density is as a measure of the uniqueness of the data in a column: high density --> less unique data. Density values range from 0 to 1.0.  There are different (but equivalent) ways to think of density.

Density = 1/[# of distinct values in a column]
Density = Avg. number of duplicates for a given value / Total row count

Consider a Customers table with a CountryID column and a RegionID column. Suppose that 100 different CountryID values are present in the table, and 1000 different RegionIDs are present. The density of the CountryID column would be higher than the density of the RegionID column because a typical country is less unique than a typical Region. Note that this is an overall characterization of a range of values. There may be some regions that occur more frequently than some countries -- a U.S.-based company might have more customers in California than in Estonia, for example.  But CountryID still has higher density than RegionID simply because the average country is represented in more rows than the average region.

Density of the CountryID column:

Density = 1/[# of distinct values in the column]
Density = 1/100 = 0.01

Note that the density of 0.01, or 1%, corresponds to the percentage of the total rows that would be returned by a query for a single value.  In other words, an average country makes up 1% of the table.  if there were 10,000 rows in the table, there were be 100 rows (1% of 10,000) per country, on average.  

Density of the RegionID column:

Density = 1/[# of distinct values in a column]
Density = 1/1000 = 0.001

The density of the CountryID column (0.01) is higher than the density of the RegionID column (0.001) because the CountryID column is less unique.

Q: What is the density of a column that contains only a single value repeated in every row?
A: 1.0 -- the highest possible density.

Q: What is the density of a unique column, such as an IDENTITY or primary key column?
A: 1/[row count]

Hands-On Example
Here is a script to create the scenario described above. 

use tempdb
go
create table customers (customerid int identity, country varchar(10), region varchar(10))
go
set nocount on
begin tran
declare @x int
set @x = 1
while (@x <=10000)
begin
    insert into customers (country, region) values ('Country' + convert (varchar, @x % 100), 'Region' + convert (varchar, @x % 1000))
    set @x = @x + 1
end
while @@trancount > 0 commit tran
go
create statistics stats_country on customers (country) with fullscan
create statistics stats_region on customers (region) with fullscan
create statistics stats_country_region on customers (country, region) with fullscan
go
dbcc show_statistics (customers, stats_country)
dbcc show_statistics (customers, stats_region)
go

Run this and check out the DBCC SHOW_STATISTICS output for the stats on the [country] and [region] columns.  In the SHOW_STATISTICS output you'll see a value called "All density".  This should match the values predicted in the calculations described above: Country and Region densities of 0.01 and 0.001, respectively.  (TIP: you should generally use "All density" and disregard the "Density" value in SHOW_STATISTICS's first resultset -- the value in the "Density" column is generally not used by the query optimizer.)  If the statistics are multi-column as in the [stats_country_region] statistics created in this example, an All density value is also maintained for combinations of columns; this can help the optimizer make more accurate predictions when a query includes a filter on more than one column.

Predicate
A predicate is an expression that evaluates to True or False. In SQL you encounter predicates in joins and in WHERE or HAVING clauses, where they are used to either filter out or qualify rows. Here's a query with two filter predicates:

SELECT *
FROM table1
WHERE column1 > 20 AND column2 IS NULL

And a query with one join predicate:

SELECT *
FROM table1
INNER JOIN table2 ON table1.column1 = table2.column1

Selectivity
Selectivity is also a measure of uniqueness, of sorts. High selectivity implies high uniqueness, or a low number of matching values. Low selectivity implies a low uniqueness, or a high percent of matches. 

Selectivity is most commonly used to describe a predicate. Returning to the example used to describe density, consider a Customers table with a Region column. Suppose that most of the company's customers are in RegionA, and only a few are in RegionB. The predicate "Region='RegionB'" would be described as more selective because it returns a smaller percentage of the rows in the table. (As an aside, this kind of uneven distribution of data within a column is an example of data skew.)  The estimated selectivity of a predicate is essential when the optimizer is evaluating possible plans. It will affect things like join order (all other things equal, the most selective join operation is generally performed first) and whether SQL chooses to do a table scan or an index seek followed by a bookmark lookup.

Selectivity for a filter predicate against a base table can be calculated as "[# rows that pass the predicate]/[# rows in the table]".  If the predicate passes all rows in the table, its selectivity is 1.0.  If it disqualifies all rows, its selectivity is 0. (This can be confusing.  Note that 0.000001 reflects a high selectivity even though the number is small, while 1.0 is low selectivity even though the number is higher.)

The optimizer often uses the histogram steps in column statistics to estimate the selectivity of a predicate.  Re-run the sample script above, and note the third and final resultset in the "DBCC SHOW_STATISTICS(customers, stats_country)" output.  A single row in this resultset represents a step in a histogram.  Each histogram step summarizes the density of all rows in the table that have a [country] value that is equal to or less than that row's RANGE_HI_KEY value, but greater than the RANGE_HI_KEY of the preceding histogram step. The snippet below shows the first 4 rows.

Take the third row in the histogram as an example. This step summarizes all rows with a country value greater than 'Country1' (the RANGE_HI_KEY for the preceding step), but less than or equal to 'Country11' (this step's RANGE_HI_KEY).  There are 100 rows with exactly the value 'Country11' (EQ_ROWS), and 100 rows with some other value in this range (RANGE_ROWS).   The DISTINCT_RANGE_ROWS value of 1 tells us that those 100 other RANGE_ROWS are made up of exactly one other distinct [country] value.  (These happen to be the rows with a country value of 'Country10', although that information isn't present in the statistics.)  I've often thought that the DISTINCT_RANGE_ROWS column was poorly-named; a better name might be "DISTINCT_RANGE_VALUES" because it reports a count of distinct values, not rows.  The AVG_RANGE_ROWS column reports the average number of rows (100) for each of the (1) distinct values in the range.  Note that by dividing RANGE_ROWS by DISTINCT_RANGE_ROWS, you calculate step density, a measure of the uniqueness of an average value within the range of values described by that histogram step.

Some simple examples might help illustrate how the optimizer uses this data to estimate selectivity.  For the query below, the optimizer's task is to guess the selectivity of a predicate like "country='Country11'".  It will look for the "Country11" value in the histogram on the country column, and will find that this value happens to be one of the RANGE_HI_KEY values (histogram step endpoints).   As a result, it can use EQ_ROWS -- the number of rows observed with this exact value. The estimated selectivity of the predicate [country]='Country11' is 100/10,000 (EQ_ROWS / [table rowcount]), or 0.01.  The actual selectivity is also 0.01, meaning that the optimizer's estimated selectivity turned out to be accurate. 

For the query below the search value 'Country10' is not a RANGE_HI_KEY -- instead, the value falls somewhere in the middle of a range of values that is summarized by a histogram step.  This means that the optimizer will need to use step density, which implies an assumtion that the search value is typical of the values in its histogram step. The estimated number of rows with a given value in this historgram step is AVG_RANGE_ROWS, which is 100 for this histogram step.  That means that the estimated and actual selectivity are both 0.01 (100 / 10,000). 

Here's a more interesting example. Note that the optimizer's guessed selectivity is wrong: it thought that 100 rows would be returned (selectivity of 0.1 * 10000 rows in the table), but in reality no rows passed the filter predicate. The search value 'Country10xyz' falls within the same histogram step used in the prior query, so the optimizer guessed the number of rows that would be returned using the exact same calculation as in the prior example. This isn't a bug; it's just a reflection of the fact that statistics can only provide an imperfect summary of the data in a table. You can read more about this here.

The information exposed by DBCC SHOW_STATISTICS can give you a better understanding of why the optimizer thinks that a particular plan will be cheaper than other plans. This is just a quick overview; for a much more detailed description SHOW_STATISTICS output and SQL statistics in general, check out the whitepaper "Statistics Used by the Query Optimizer in Microsoft SQL Server 2008" @ https://msdn.microsoft.com/en-us/library/dd535534(v=sql.100).aspx.  (If you haven't ever read this paper, it's a very good read.)

Cardinality
All of this brings us finally to cardinality, which is kind of "where the rubber hits the road" when you are trying to understand why a query plan is slow.

For our purposes, cardinality can be thought of as the number of rows returned by a query operator.  (A few examples of physical query operators that you will probably recognize: Index Seek, Nested Loop Join, Filter)  The cardinality of an operator like a Filter is determined by multiplying the selectivity of that operator -- that is, the % of its input rows that it will pass -- by the cardinality (rowcount) of the operator's child.  If an operator receives 500 rows from its child operator and has a selectivity of 0.3 (30%), it has a cardinality of 500*0.3, or 150 rows.

Each operator in a query plan has an estimated cardinality (the number of rows the optimizer guessed that the operator would return) and an actual cardinality (the number of rows that the operator returned in the real world).  You can see both by running a query with "SET STATISTICS PROFILE ON".

Sometimes the query optimizer cannot accurately predict the number of rows that a given operator will return. This can prevent SQL from estimating the cost of a query plan correctly, which can lead to the selection of a suboptimal plan. Cardinality estimation errors are one of the most common causes of slow query plans in SQL Server, so it is very important to know how to identify cardinality estimation problems in a query plan. (That's a little beyond the point of this post, though -- here I'm just defining some terms.) In the plan above, you can see that SQL made a cardinality estimation error: it guessed that the filter would be very selective and would only pass 21 of 2971 rows, when in reality almost all of the rows passed the filter.

The optimizer has a number of ways to estimate cardinality, none of which are completely foolproof. 

1. If the predicate is simple like "column=123" and if the search value happens to be a histogram endpoint (RANGE_HI_KEY), then EQ_ROWS can be used for a very accurate estimate.
2. If the search value happens to fall between two step endpoints, then the average density of values in that particular histogram step is used to estimate predicate selectivity and operator cardinality.
3. If the specific search value is not known at compile time, the next best option is to use average column density ("All density"), which can be used to calculate the number of rows that will match an average value in the column. 
4. In some cases none of the above are possible and optimizer has to resort to a "magic number"-based estimate.  For example, it might make a totally blind guess that 10% of the rows will be returned, where the "10%" value would be hardcoded in the optimizer's code rather than being derived from statistics. 

In most scenarios option #1 is the ideal, but because most values aren't histogram endpoints, option #2 is the most frequently-used method for simple filter and join predicates. Option #4 usually provides the worst estimates, and is only used when there is no other alternative. Thankfully, most of the time when a magic number is used, this is a side effect of poor query design; by rewriting the query you can enable a more accurate cardinality estimate based on either average column density or histogram lookup.

We could just store the external key in the database table and use it to locate the correct widget, but we couldn’t afford to scan the widget table for every lookup. The number of rows in the widget table was expected to be large, so we needed the lookup to be indexed. Unfortunately, SQL has a limit of 900 bytes for the keys in each index, and the external key values can be longer than this. If you create an index on combination of columns that could contain more than 900 bytes, you get this warning:

Warning! The maximum key length is 900 bytes. The index 'xyz' has maximum length of 4000 bytes. For some combination of large values, the insert/update operation will fail.

Then if you try to insert a row into the table that has more than 900 bytes in the indexed columns, it will fail with this error:

Msg 1946, Level 16, State 3, Line 11

Operation failed. The index entry of length 1501 bytes for the index 'xyz' exceeds the maximum length of 900 bytes.

Even if this restriction within SQL didn’t exist, I wouldn’t ever think of creating a table with a >1KB primary key. A surrogate primary key like an IDENTITY column or a GUID/UNIQUEIDENTIFIER column should be used, instead. You'd use the surrogate as your primary key, and all lookups and foreign keys within the data tier should use the surrogate. But the external system doesn’t know anything about the surrogate keys used in the data tier; the only identifier that the external system can pass to the database is that >900 byte external key value. So while a surrogate for primary key is a good idea here, it doesn’t solve the immediate problem – we still need to be able to do an efficient (indexed) lookup of an external key value in the database, even though a column that could store the external key values would be too large to index.

If you find yourself in a similar situation, here are some alternatives to consider:

A. Change the external system to use an identifier that is always less than 900 bytes long. Examine the entity’s attributes: is there some combination of attributes that together make up a candidate key (in other words, is there some combination of other attributes that is guaranteed to be unique for each entity instance)?  Even if the external system doesn’t internally use those columns as an identifier, it could still pass them to the data tier when it wanted to look up a widget.  In the database, you’d create a UNIQUE index on this column or combination of columns.  

B. Require non-key columns as additional criteria for each lookup. For example, suppose a widget has a “Name” attribute that is always short enough to be indexed. Name is not guaranteed to be globally unique, so neither the external system nor the relational model can use this as a key. But if the column is fairly selective it may be sufficient to give you lookups that are efficient enough that it’s almost like doing an index seek on the external key. In this example you would index the [name] column in the database, and use a routine for lookups like this.

   CREATE PROC usp_get_widet @external_key VARCHAR(MAX), @name NVARCHAR(60) AS
   SELECT *
   FROM widgets
   WHERE external_key = @external_key AND name = @name;

SQL will use a seek on the non-clustered [name] index to narrow the results down to a tiny percentage of the table (hopefully no more than a few rows), then it will do a residual filter to get rid of all but the single row that matches the external key value. Note that you can add more than one supplemental search column if necessary to increase the selectivity of the non-key index seek. Also be sure to note that you still need to do the filter on the [external_key] column. That predicate won’t be satisfied by an index seek, but it’s still necessary for correctness. Consider making the full external key INCLUDEd in the nonclustered index, to minimize the number of bookmark lookups the engine needs to do to find the single matching row.

One obvious downside to this approach is that the additional required search criteria complicate the interface between the database and the external component.

C. Truncate the external key to 900 bytes and index the truncated portion of the key. For example, your widget table might look like this:

   CREATE TABLE widgets (
       surrogate_key INT PRIMARY KEY IDENTITY,
       external_key VARCHAR(MAX),
       external_key_fragment AS LEFT (external_key, 900),
       ...

You’d create a (nonunique) index on the [external_key_fragment] column, and your lookup procedure might look something like this:

   CREATE PROC usp_get_widet @external_key VARCHAR(MAX) AS
   SELECT *
   FROM widgets
   WHERE external_key = @external_key
       AND external_key_fragment = LEFT (@external_key, 900);

Of course, this assumes that some predictable 900-byte portion of the external key is reasonably selective. If it’s possible for a large portion of the rows to have the same initial 900 bytes (assuming the fragment is generated in the way shown above), then this won’t help any – the optimizer will correctly estimate that a scan would be more efficient than a seek, and you’ll still end up scanning the entire table for each lookup.

D. Break up or parse the external key into its component parts, and index a selective subset of the parts.  This assumes that the long external key is really a serialized version of a composite key. It must have some underlying structure, and it must be practical to break the key into its child parts within the data tier. It also assumes that some portion of this structured key is guaranteed to be unique enough to support an efficient index seek that only returned a few rows.

For example, suppose that the external key was string that actually encoded a hierarchy like “RootIdentifierX\ChildIdentifierY\GrandChildIdentifierZ\...”  If you’re willing to push knowledge of the structure of this serialized key down into the data tier, the database could parse this apart and use the identifiers for the first few levels of the hierarchy to do a secondary lookup that could be satisfied via an index seek. You’d need to store those parts of the external key in separate columns (so they could be index) in addition to the full external key.  Once you’ve indexed the portions that you can, consider adding the remainder as an INCLUDEd column in the index; that would allow the index seek to locate the single matching row without resorting to a bookmark lookup. (It would also make it less likely that the optimizer would choose to scan the table because it guessed that a seek on the component parts wouldn’t be selective enough.)

E. Hash the external key and index the hash. This is essentially building a hash table and persisting it in the database. With this approach, your widgets table might look something like this:

 CREATE TABLE widgets (
     surrogate_key INT PRIMARY KEY IDENTITY, 
     external_key VARCHAR(MAX), 

     -- A computed column that hashes the external key (MD5 returns a 
     -- 128-bit hash). 
     external_key_hash AS CAST (HASHBYTES('MD5', external_key) AS VARBINARY(16)), 
     ...

The [external_key_hash] column would be indexed, and the lookup proc would do this:

 CREATE PROC usp_get_widet @external_key VARCHAR(MAX) AS 
 SELECT * 
 FROM widgets 
 WHERE external_key = @external_key 
     AND external_key_hash 
          = CAST (HASHBYTES('MD5', @external_key) AS VARBINARY(16));

It could be argued that the CAST(HASHBYTES(…)) stuff should be hidden within a UDF (ditto for alternatives (C) and (D)):

   CREATE FUNCTION dbo.fn_hash_external_key (@urn nvarchar(max))
   RETURNS VARBINARY(16)
   WITH SCHEMABINDING
   AS
   BEGIN
      RETURN HASHBYTES ('MD5', @urn);
   END;

The computed column and the lookup proc would both reference this function so that the details of the hash function lived in a single place. That would indeed be cleaner, but before taking that route be sure you’re aware of the potential problems with scalar UDF performance that are described at https://www.developerfusion.com/community/blog-entry/8389063/tsql-scalar-functions-are-evil/ and https://sqlblog.com/blogs/adam_machanic/archive/2006/08/04/scalar-functions-inlining-and-performance-an-entertaining-title-for-a-boring-post.aspx. 

Of all of these, option (A) is the only one that I find very aesthetically pleasing.  The others could work in a pinch, though.  Please drop me a comment if I’m overlooking some other alternative here.

(Cross-posted here)

Query Hash Statistics is a free download that collects historical query and query plan fingerprint statistics and allows you easily to view the data in SSMS and see the true cumulative cost of the queries in each of your databases.  You can use the project as-is, or – if you’re the tinkering type – disassemble it and build your own custom query hash stats solution. (Source code is included.)  

Download Query Hash Statistics @ https://code.msdn.microsoft.com/QueryHashStatistics 

Project Description:

Query Hash Statistics collects Microsoft SQL Server query execution statistics and uploads the data to a central warehouse. The solution also includes a set of reports to analyze and visualize this data within SQL Server Management Studio (SSMS). Query Hash Statistics differs from most other query cost analysis tools in that it builds on the "query fingerprint" and "query plan fingerprint" features that were added in SQL Server 2008. These features make it possible to calculate the cumulative cost of all executions of a query, even when the application doesn't parameterize its queries. A solution like Query Hash Statistics that builds on these features can do low-overhead query cost monitoring with a degree of accuracy that was impossible in prior releases of SQL Server.

Because the solution relies on new SQL engine features that were added in SQL Server 2008, it only works with SQL Server 2008 and later.

Source code is included, to demonstrate how to build similar solutions. In addition to highlighting the engine's query and query plan fingerprint features, the project shows how to use extensibility features in Data Collector and SSMS. By following the patterns demonstrated in this project, you can learn how to:  

• Identify cumulative query cost without a profiler trace using the SQL Server "plan fingerprint" and "query fingerprint" features.
• Use Data Collector to upload data from SQL DMVs to a central management database.
• Build reports that can be hosted within SQL Server Management Studio (SSMS) to analyze the data in your central management database.

Check it out, and let us know about your experiences in the Discussions tab on the MSDN Code Gallery project. If you have a suggestion or think you've found a bug, please open a new issue under the Issue Tracker tab. Thanks!

Additional query keywords: QueryHashStats Query Hash Stats query performance query troubleshooting cumulative query cost analysis Microsoft SQL Server MSSQL MS SQL T-SQL TSQL Transact-SQL query plan cost cumulative plan cost find expensive queries find slow queries DMV dm_exec_query_stats plan fingerprints query fingerprints plan hash query hash query_plan_hash query_hash dm_exec_requests top queries most expensive queries

rpt_query_hash_thumbnails.png

Let me start this post off by saying that I understand why user-defined data types (UDTs) in T-SQL seem alluring.  At first glance, you might expect UDTs to give you some very nice benefits, such as:

Expected Benefit #1: Suppose you have a field that stores percentages values (0-100).  In T-SQL you might choose the tinyint data type for such a field, but this isn’t the only data type that you could have chosen to store percentage values.  If there is more than one table that contains a percentage-type field, you could end up with the same sort of value being stored using different data types.  By creating a UDT called PERCENT, you can avoid the need for database developers to memorize “Percentages should be stored using tinyint.”

Reality: This is based on the idea that it’s easier for a developer to memorize “Percentages are stored using a UDT named PERCENT” than it is to memorize “Percentages are stored using system type tinyint.”  The problem is that you have to know the underlying data type in order to deal with the column correctly.  For example, a database developer that was retrieving a PERCENT (UDT) field in a VB.NET or C# app would need to know what .Net data type should be used to retrieve values from the field.  A thoughtful dev would wonder: “Maybe I should store this PERCENT value in a byte variable, which can hold values from 0 to 255.  But wait: Can a PERCENT column hold a negative percentage?  If so, I’ll need to use the short data type, which can represent negative integers.  Hold on a second!  I wonder if this PERCENT data type can store fractional percentages?  If so, I’d need to retrieve the column value into a decimal variable...” 

In other words, if you don’t use a UDT, the application developer has to memorize “Percentages are stored using system type tinyint.”  But if you use a T-SQL UDT the application developer has to know both “Percentages are stored using a UDT named PERCENT” and “Percentages are stored using system type tinyint.”  The attempt to make the app developer’s life easier can backfire and actually made it a little harder.  (It’s worth mentioning that SQLCLR UDTs don’t have this same limitation.  App developers can reference an assembly that gives them a client-side data type that corresponds to the server-side SQLCLR UDT, so the developer doesn’t have to know the details of the UDT’s implementation in order to use it.)

Expected Benefit #2: A tinyint can store 0-255, but your percentage fields should only allow 0-100.  You can restrict a column’s domain by defining a CHECK constraint on the column.  But if there are several percentage type columns in several different tables, you’d need to define identical CHECK constraints on each column.  Wouldn’t it be nice if you could attach the constraint to the PERCENT user-defined data type, so that it would apply automatically to each column that used this UDT? 

Reality: This would be a real benefit of T-SQL UDTs.  Unfortunately, the ability to attach a rule to a UDT has been officially deprecated, so you shouldn’t use this functionality.  Since you can no longer attach custom domain constraints or other business rules to the data type, a T-SQL UDT is really nothing more than an alias for a system data type.  Sorry!

Expected Benefit #3: Suppose that your requirements have changed, and now you need to be able to store fractional percentages.  You decide to store percentages as decimal(3,2) instead of tinyint.  If all percentage columns were defined as tinyint, you’d have to go around to all of the columns and modify each column’s data type separately.  And to change a column’s data type you must first drop any indexes or constraints that reference the column.  What a pain.  Wouldn’t it be easier if you could just change the PERCENT UDT definition in one place?

Reality: T-SQL doesn’t have an ALTER TYPE command for UDTs.  To change a UDT definition you must do a DROP TYPE followed by CREATE TYPE.  But of course you can’t drop a type that is being used in a column, so you have to do an awkward little dance to switch all of the columns that use the UDT to use the underlying system data type, instead (so they no longer reference the UDT), then drop and redefine the UDT, then ALTER the columns a second time to re-reference the new UDT. 

                ALTER TABLE tbl1 ALTER COLUMN foo_pct TINYINT;

                ALTER TABLE tbl2 ALTER COLUMN foo_pct TINYINT;

                ALTER TABLE tbl3 ALTER COLUMN foo_pct TINYINT;

                ALTER TABLE tbl4 ALTER COLUMN foo_pct TINYINT;

                DROP TYPE PERCENT;

                CREATE TYPE PERCENT FROM DECIMAL(3,2);

                ALTER TABLE tbl1 ALTER COLUMN foo_pct PERCENT;

                ALTER TABLE tbl2 ALTER COLUMN foo_pct PERCENT;

                ALTER TABLE tbl3 ALTER COLUMN foo_pct PERCENT;

                ALTER TABLE tbl4 ALTER COLUMN foo_pct PERCENT;

And if any indexes or constraints reference any of these columns, you’d still have to drop those before the first ALTER COLUMN, and recreate them afterward.  If you have to write DDL that redefines the columns using the base data type that the UDT is supposed to hide, where’s the abstraction benefit?  More importantly, if you didn’t use UDTs this same change can be done with less than half the code:  

                ALTER TABLE tbl1 ALTER COLUMN foo_pct DECIMAL(3,2);

                ALTER TABLE tbl2 ALTER COLUMN foo_pct DECIMAL(3,2);

                ALTER TABLE tbl3 ALTER COLUMN foo_pct DECIMAL(3,2);

                ALTER TABLE tbl4 ALTER COLUMN foo_pct DECIMAL(3,2);

So there you have the reasons why I avoid using T-SQL user-defined types in my own code.  It’s not that UDTs are all that bad; it’s just that in my experience they fall short of their promise, and they can make some things even more cumbersome than the same task would be without the use of UDTs.  If you think I’m overlooking some concrete benefit of UDTs, I’d love to hear from you in the comments section.  If you do choose to use UDTs, just be sure you aren’t basing the decision to use them on inaccurate assumptions about the benefits they’ll provide. 

An important caveat: The above applies to simple T-SQL UDTs.  You can also create a user-defined type in SQL Server using SQLCLR and .Net.  While a T-SQL UDT is never anything more than a simple alias for one of the system data types, a SQLCLR UDT can be a truly custom data type.  With SQLCLR you can implement complex types, embed custom business logic like domain constraints or validation rules into the type, and so on.  In other words, with SQLCLR UDTs you can actually realize the benefit that I called “Expected Benefit #2”, above.  As mentioned earlier, you also get much of Expected Benefit #1.  SQLCLR UDTs still have the limitations described in #3, but the ability to extend SQL Server in the way that SQLCLR allows is very powerful.  That benefit could easily compensate for the issue discussed in #3, if the task you are trying to accomplish would be much harder to implement using one of the system data types. 

When you're trying to size a server that will be running a well-known application -- say, SAP, or Exchange -- you have a chance of getting a recommendation that is reasonably precise.  This is because the person or tool that is making the sizing recommendation can make well-educated guesses based on knowledge of the way that users generally use that app, and the app's typical transaction costs.  But if you want non-app-specific guidance for an unknown or arbitrary database app, the problem gets much harder.  This is because a single business transaction from one application (say, “log in”, or “submit purchase order”) could be orders of magnitude cheaper or more expensive than the exact same type of business transaction in a different app.  In other words, there are limits to how precise application-independent sizing guidance can be for an unknown app when the inputs are nothing more than simple metrics like # of concurrent users, queries/second, or transactions/sec. 

Application-Specific Server Sizing

My first recommendation is to go to the application developer, not the RDBMS developer, for the most meaningful sizing guidance.  I don't think that even Microsoft has explicit sizing guidance for every MS app that runs on SQL, but there are sizing tools or whitepapers that make hardware recommendations for many or most of our SQL apps.  A few examples:

• Sharepoint
• Reporting Services
• Forefront
• Team Foundation Server

If you can't find explicit sizing guidelines for the application you'll be running on SQL, I'd try these alternatives, listed in no particular order:

• Case studies from the ISV (case studies often include server hardware details and information about concurrent user counts, data volumes, etc)
• Post a query to online forums dedicated to the app (for Microsoft apps you might try the TechNet forums)
• Contact the app ISV directly

Application-Independent Server Sizing

You may come up empty-handed in your search for application-specific sizing guidance.  If the application was developed in-house or if you're sizing a server for an app that doesn't even exist yet, you'll definitely be in this situation.  Application-independent hardware sizing guides do exist.  The first type of resource in this category may not be as simple to use as you might like, but the conclusions you'll come to will be more trustworthy.  The first two whitepapers below take the approach of giving you the tools you need to identify the key characteristics of an existing application workload, then providing you with a process to translate that into a set of hardware requirements.  The database sizing tool will give you a fairly accurate estimated database size, but to get it you must provide a lot of schema details.  These are not simple processes, so don't expect to have a set of server specs in 30 seconds.  But the complexity is inherent to the problem and can't be eliminated without sacrificing precision, so you can consider it time well spent. 

• Disk subsystem sizing guide from the SQL CAT team: https://msdn.microsoft.com/en-us/library/ee410782(SQL.100).aspx
• Fast Track Data Warehouse 2.0 Architecture: https://msdn.microsoft.com/en-us/library/dd459178.aspx (not app-specific, but tailored to a particular class of application)
• DataSizer tool (helps estimate database sizes based on your table schema):  https://www.microsoft.com/downloads/details.aspx?FamilyID=564c5704-d4f5-4ee8-9f3c-cb429499d075&displaylang=en – This was written for SQL 7.  The estimates won’t be quite as accurate for more recent SQL versions, but in most cases they should be pretty close unless you’re using Enterprise Edition SQL features like compression.  (For details about performing this calculation by hand in SQL 2008 R2, see the "Estimating the size of a database" topic in Books Online.) 

Finally, there are SQL sizing tools that will give you hardware recommendations for an unknown app with little to no time investment on your part.  These might provide a nice starting point, but I’d take their recommendations with a grain of salt.  As mentioned earlier, you can't estimate required hardware with any real precision when the app and the nature of its SQL workload are unknown variables, so be cautious about any recommendation that is made on the basis of a few vague metrics. 

• Dell SQL Advisor: https://www.dell.com/content/topics/global.aspx/tools/advisors/sql_advisor?c=us&cs=555&l=en&s=biz 
• HP SQL Sizers (note: separate links to sizing tools for BI workloads vs. OLTP workloads): https://h71028.www7.hp.com/enterprise/cache/3887-0-0-0-121.html

TRUNCATE When You Can

The first, and simplest, best practice is to use TRUNCATE instead of DELETE when possible.  TRUNCATE will only log extent or page deallocation operations, while a DELETE is a row-by-row operation and must log the pre-delete image of every deleted row.  TRUNCATE runs nearly instantaneously, even on very large tables that might take days or hours to clear out with a DELETE statement.  But of course TRUNCATE can only be used when you need to delete all of the data in a table. 

Consider Partitioning

If you have Enterprise or Datacenter Edition and are running SQL Server 2005 or later, you may be able to partition your data.  With an appropriately-selected partitioning scheme, you can simply switch out the old data.  The cost of this is generally similar to a TRUNCATE or DROP TABLE – in other words, nearly instantaneous.  You can learn more about SQL Server partitioning in this whitepaper by Kim Tripp. 

Delete in Chunks

Most of the time you only want to delete a subset of the rows in a table.  If you can’t use partitioning, you must use DELETE.  Suppose that you needed to delete a lot of data with a straightforward DELETE statement like this one:  

    DELETE FROM big_table 
    WHERE ready_to_archive = 1;

 This could cause two problems.  The first is that the DELETE will acquire exclusive locks on all deleted rows, and those locks will be held until the DELETE completes.  If the DELETE runs for hours, other users trying to query or modify the table may be blocked for hours.  If more than a few thousand locks are acquired, SQL may escalate to an X table lock that will prevent anyone else from modifying any row in the table, even rows that will not be deleted.  

 The second issue that you might see is transaction log growth.  If your database is in simple recovery mode, under normal circumstances a relatively small .LDF file will be sufficient because the space in the log file is frequently reused.  But SQL can only reuse space in a transaction log prior to the region that records the oldest uncommitted transaction.  The long-running DELETE may cause your .LDF file(s) to grow.  If the file grows too large or if you have disabled autogrow you will eventually run out of log space, bringing all activity in the database to a halt. 

  You can avoid both of these problems by deleting the rows in chunks, like this:

    DECLARE @rows_affected BIGINT;
    DECLARE @delete_batch_size INT;
    SET @delete_batch_size = 1000;
    SET @rows_affected = @delete_batch_size;

    WHILE (@rows_affected = @delete_batch_size)
    BEGIN
        DELETE TOP (@delete_batch_size) FROM big_table 
        WHERE ready_to_archive = 1;
        SET @rows_affected = @@ROWCOUNT;
    END;

 This will delete 1000 rows at a time, continuing until there are no more rows to delete.  Each batch of 1000 rows is deleted in a separate transaction, so the batch can run for a long time without holding locks for a long time or causing transaction log growth.  You can experiment with different batch sizes, but in my experience a batch size above 1000 rows doesn’t improve performance that much. 

 If you want to avoid transaction log growth but don’t care about reduced concurrency (for example, if the DELETE is running during a maintenance window when no one else will need to query the table), you may see a small performance boost by deleting in chunks but forcing a table lock on each DELETE.  The cost of lock management and other housekeeping tasks is reduced when SQL doesn’t have to worry about concurrency.  A TABLOCK hint also allows SQL to reclaim empty pages that would otherwise remain allocated to the table. 

     ...
    WHILE (@rows_affected = @delete_batch_size)
    BEGIN
        DELETE TOP (@delete_batch_size) FROM big_table WITH (TABLOCK)
        WHERE ready_to_archive = 1;
        SET @rows_affected = @@ROWCOUNT;
    END;

 On the other hand, if you don’t care about transaction log growth but you do want to prevent an unnecessary table lock escalation, you can use trace flag 1224:  

    DBCC TRACEON (1224); 

    DELETE FROM big_table 
    WHERE ready_to_archive = 1;

     DBCC TRACEOFF (1224);

 Trace flag 1224 is documented and supported.  It prevents SQL from escalating to table locks except when it must because the server is running low on memory.  This may also be a good compromise solution if you want to maximize concurrency but you require that the entire delete operation be transactional. 

Consider Indexes and Foreign Keys

If your DELETE statement is running slower than you expect, it might make sense to drop indexes on the table and/or any foreign keys referring to the table, do the DELETE, then recreate the objects you dropped.  Indexes on a table can dramatically slow down a large DELETE, even to the point where it is cheaper to rebuild the index from scratch after the DELETE than it would be to maintain the index in place.  (Note that sometimes you clearly will see the extra index maintenance operators in the query plan, and other times the query plan won’t show this work.  But just because you don’t see any Index Delete operators in the plan doesn’t mean that the DELETE’s performance isn’t affected by index maintenance overhead.  For details, see my earlier post on Wide vs. Narrow Plans.)  Foreign keys on other tables that refer to the table that you’re deleting from can also slow down the DELETE.  The FK overhead comes from the fact that SQL must verify that you’re not deleting any rows that are referenced by rows in the child table.  

 It’s difficult to generalize about when it might be cost effective to drop indexes or FKs.  The effect on the overall runtime of the delete job depends on how many indexes are on the table, the size of the indexes, whether any FK columns in other tables are indexed, the portion of the table you’re deleting, whether there is correlation between nonclustered and clustered index keys values, etc.  Personally, I’d consider testing out the approach and quantifying the benefits when all of the following are true:

1. Your DELETE query runs slower than you would like
2. Temporarily dropping indexes or FKs won’t hurt anything (for example, if the DELETE runs in a maintenance window when no one else is using the database)
3. You are deleting a fairly large % of the rows in the table (rule of thumb: > 10% of the rows) 

      https://weblogs.sqlteam.com/dang/archive/2008/02/21/Dont-Bloat-Proc-Cache-with-Parameters.aspx

      https://weblogs.sqlteam.com/dang/archive/2008/02/18/Why-Parameters-are-a-Best-Practice.aspx

Below is a handy DMV query that can quickly identify which queries have many different plans cached for the same query.  In most cases, this indicates a lack of explicit parameterization of the query.  This uses the query plan hash/query plan fingerprint feature that I wrote about in an earlier post.  It relies on the fact that two queries that have different inline literal values (e.g. "SELECT...WHERE col1 = 123", vs. "SELECT...WHERE col1 = 456") will get two different query plans, but the plans will have the same query_hash value.  You must be running SQL Server 2008 or later to use this.   

 WITH duplicated_plans AS (
    SELECT TOP 20 
        query_hash,
        (SELECT TOP 1 [sql_handle] FROM sys.dm_exec_query_stats AS s2 WHERE s2.query_hash = s1.query_hash ORDER BY [sql_handle]) AS sample_sql_handle,
        (SELECT TOP 1 statement_start_offset FROM sys.dm_exec_query_stats AS s2 WHERE s2.query_hash = s1.query_hash ORDER BY [sql_handle]) AS sample_statement_start_offset,
        (SELECT TOP 1 statement_end_offset FROM sys.dm_exec_query_stats AS s2 WHERE s2.query_hash = s1.query_hash ORDER BY [sql_handle]) AS sample_statement_end_offset,
        CAST (pa.value AS INT) AS dbid,
        COUNT(*) AS plan_count 
    FROM sys.dm_exec_query_stats AS s1
    OUTER APPLY sys.dm_exec_plan_attributes (s1.plan_handle) AS pa 
    WHERE pa.attribute = 'dbid'
    GROUP BY query_hash, pa.value
    ORDER BY COUNT(*) DESC
)
SELECT
    query_hash,
    plan_count,
    CONVERT (NVARCHAR(80), REPLACE (REPLACE (
        LTRIM (
            SUBSTRING (
                sql.[text],
                (sample_statement_start_offset / 2) + 1,
                CASE
                    WHEN sample_statement_end_offset = -1 THEN DATALENGTH (sql.[text])
                    ELSE sample_statement_end_offset 
                END - (sample_statement_start_offset / 2)
            )
        ),
        CHAR(10), ''), CHAR(13), '')) AS qry,
    OBJECT_NAME (sql.objectid, sql.[dbid]) AS [object_name],
    DB_NAME (duplicated_plans.[dbid]) AS [database_name]
FROM duplicated_plans
CROSS APPLY sys.dm_exec_sql_text (duplicated_plans.sample_sql_handle) AS sql
WHERE ISNULL (duplicated_plans.[dbid], 0) != 32767 -- ignore queries from Resource DB
AND plan_count > 1;