<?xml version="1.0" encoding="utf-8"?>
<rss xmlns:atom="https://www.w3.org/2005/Atom" xmlns:sy="https://purl.org/rss/1.0/modules/syndication/" xmlns:dc="https://purl.org/dc/elements/1.1/" xmlns:content="https://purl.org/rss/1.0/modules/content/" xmlns:wfw="https://wellformedweb.org/CommentAPI/" xmlns:slash="https://purl.org/rss/1.0/modules/slash/"
  version="2.0">
  <channel>
    <title>The PFE Chronicles</title>
    <atom:link
      href="https://docs.microsoft.com/archive/blogs/dataplatform/feed.xml"
      rel="self"
      type="application/rss+xml" />
    <link>https://docs.microsoft.com/archive/blogs/dataplatform/feed.xml</link>
    <description>Some advice and experience for you from SQL Server PFEs</description>
    <lastBuildDate>Wed, 17 Oct 2018 09:10:05 GMT</lastBuildDate>
    <language>en-US</language>
    <sy:updatePeriod>hourly</sy:updatePeriod>
    <sy:updateFrequency>1</sy:updateFrequency>
    <item>
      <title>Variables, Literals, Procs and Stats</title>
      <link>https://docs.microsoft.com/archive/blogs/dataplatform/variables-literals-procs-and-stats</link>
      <pubDate>Sat, 07 Jul 2018 08:00:36 GMT</pubDate>
      <dc:creator><![CDATA[Chris Lound]]></dc:creator>
      <guid
        isPermaLink="false">https://blogs.technet.microsoft.com/dataplatform/?p=6445</guid>
      <description><![CDATA[This post by US PFE Susan Van Eyck goes through considerations when developing stored procedures...]]></description>
      <content:encoded><![CDATA[This post by US PFE Susan Van Eyck goes through considerations when developing stored procedures using local variables and literals, and their effect on the query plan / performance.

&nbsp;

I ran across a very interesting <a href="https://www.brentozar.com/archive/2014/06/tuning-stored-procedures-local-variables-problems/?utm_source=Brent+Ozar+Unlimited%C2%AE+List&amp;utm_campaign=ad4e98a3e1-RSS_EMAIL_CAMPAIGN&amp;utm_medium=email&amp;utm_term=0_8e3e861dd9-ad4e98a3e1-296081077&amp;goal=0_8e3e861dd9-ad4e98a3e1-296081077&amp;mc_cid=ad4e98a3e1&amp;mc_eid=ed318ed172">blog post</a> by Kendra Little this week about the perils of troubleshooting – or developing – stored procedure using local variables.

I’ve often done this myself.  You morph the CREATE statement into a DECLARE, plug in some values, then try to figure out why the query is misbehaving.

&nbsp;


[sql]--CREATE PROC dbo.DoSomethingUseful

DECLARE       @parm1 INT = 875,

@parm2 VARCHAR(10) = 'ABC'

AS

BEGIN

-- Useful bits here

END

GO[/sql]


&nbsp;

The problem here is that those local variables behave differently than either stored procedure parameters or literal values and you can end up with bad/wrong/misleading query plans because of it.

To demonstrate, consider these two very similar queries.  Both will return 4 rows.

&nbsp;


[sql]SELECT City FROM Person.Address WHERE StateProvinceID = 80;

DECLARE @StateProvinceID INT = 80;

SELECT City FROM Person.Address WHERE StateProvinceID = @StateProvinceID;[/sql]


&nbsp;

Executing them with the options to <b>Include Actual Query Plan</b> and <b>SET STATISTICS IO ON</b> you find that the first one (using the literal value) has a very efficient query plan (Index Seek + Lookups) requiring only 10 logical reads, but the second (using the variable) is executed using an Index Scan at a cost of 216 logical reads.  <i>[Interestingly we’re scanning a <u>nonclustered</u> index because it happens to covers the query and saves us the higher cost (346 logical reads) of a clustered index scan.]</i>

<a href="https://msdnshared.blob.core.windows.net/media/2018/04/clip_image0021.gif"><img width="732" height="423" title="clip_image002" alt="clip_image002" src="https://msdnshared.blob.core.windows.net/media/2018/04/clip_image002_thumb.gif" border="0" /></a>

&nbsp;

The first query is auto-parameterized (note the <b>=@1</b> in the plan’s SQL text), but that doesn’t account for the difference.  Adding another table to the query to increase up its complexity and prevent auto-parameterization, we get the same results.  Rather, the difference is due to which statistics are being used, which is in turn determined by whether or not we’re using a local variable.  Looking at the Estimated Number of Rows for the two plans they’re very different – 4 and 265.

<a href="https://msdnshared.blob.core.windows.net/media/2018/04/image238.png"><img width="803" height="364" title="image" alt="image" src="https://msdnshared.blob.core.windows.net/media/2018/04/image_thumb221.png" border="0" /></a>

&nbsp;

An Index Seek + Lookups is a good data access strategy when the Seek returns a relatively low number of rows and there’s an index supporting the Lookups.  That’s the case with the first plan with its (very accurate) estimate of 4 rows and Lookups into the Clustered Index.  But the Seek/Lookup pattern doesn’t scale well, and as row counts increase you hit a “tipping point” where it actually is cheaper to Scan the Clustered Index (or another index that covers the query).  That’s what’s happened in the second plan – the estimated cost of 265 Lookups (530 logical reads because the index has a depth of 2) is much higher than that of scanning the covering index (216 logical reads).

So we have a rational explanation for the differing <u>plans</u>, but how did they come to have such different estimated row counts?

When evaluating query plans options the Optimizer checked the statistics associated with the index on StateProvinceId.  There are two values in the statistics object that offer guidance on expected row counts.  One is <b>Density</b> – a measure of the uniqueness of any single column value in the context of the entire table.  If you multiply it by the number of rows in the table…

&nbsp;


[sql]DBCC SHOW_STATISTICS ('Person.Address', IX_Address_StateProvinceID) WITH DENSITY_VECTOR;[/sql]


<a href="https://msdnshared.blob.core.windows.net/media/2018/04/image239.png"><img width="534" height="114" title="image" alt="image" src="https://msdnshared.blob.core.windows.net/media/2018/04/image_thumb222.png" border="0" /></a>

&nbsp;


[sql]SELECT 0.01351351 * COUNT(*) FROM person.Address; /* 265.05398514 */[/sql]


…you get 265!  Here’s another way to arrive at 265 that might make it clearer what that value represents:

&nbsp;


[sql]SELECT COUNT( * ) / COUNT( DISTINCT StateProvinceID ) FROM Person.Address; /* 265 */[/sql]


This is the value that’s used when your code includes a <b>local variable</b>.  Its degree of accuracy will depend on how evenly data is distributed across a column’s values.  In our case, an average value of 265 rows per column value is not a very representative value.  We can see that by examining the other useful value(s) in the table – the <b>Histogram</b>:

&nbsp;


[sql]DBCC SHOW_STATISTICS ('Person.Address', IX_Address_StateProvinceID) WITH HISTOGRAM;[/sql]


<a href="https://msdnshared.blob.core.windows.net/media/2018/04/image240.png"><img width="718" height="418" title="image" alt="image" src="https://msdnshared.blob.core.windows.net/media/2018/04/image_thumb223.png" border="0" /></a>

&nbsp;

Looking at values in the <b>EQ_ROWS</b> column in this excerpt, you can see that none of them is anywhere near 265!  But that value in the row for a RANGE_HI_KEY of 80 is spot on at 4.  These are much more granular, accurate stats, and these numbers are used when your code uses literal values or stored procedure parameters.

So the bottom line is don’t test your stored procedure code using local variables – and do read Kendra’s interesting post.

One more thing <u>not</u> to do – don’t assign new values to your stored procedure parameters within the code – like I’m doing here:

&nbsp;


[sql]CREATE OR ALTER PROCEDURE dbo.GetCities

@StateProvinceID INT

AS

SET @StateProvinceID += 1;

SELECT City FROM Person.Address WHERE StateProvinceID = @StateProvinceID;

GO

EXEC dbo.GetCities @StateProvinceID = 79;

GO[/sql]


&nbsp;

The query plan to execute the stored proc performs an Index Scan, but for an entirely different reason than in our local variable example.  The estimated row count for the EXEC statement is 2636 rows.  Now look back at the stats histogram values for a value of 79 – it’s also 2636 rows.  The query plan is based on the parameter values the proc is called with, so although we’re ultimately executing the query with a search value of 80, the plan is based on a value of 79.  The recommended fix is to assign the new value to a local variable declared within the stored proc – but beware – that takes us back to the original problem!

&nbsp;

Happy Exploring,]]></content:encoded>
    </item>
    <item>
      <title>Previewing file content</title>
      <link>https://docs.microsoft.com/archive/blogs/dataplatform/previewing-file-content</link>
      <pubDate>Mon, 02 Jul 2018 09:00:55 GMT</pubDate>
      <dc:creator><![CDATA[Jonathan Allen msft]]></dc:creator>
      <guid
        isPermaLink="false">https://blogs.technet.microsoft.com/dataplatform/?p=6545</guid>
      <description><![CDATA[Windows Explorer is a common tool for anyone working in the Windows environment and sometimes being...]]></description>
      <content:encoded><![CDATA[Windows Explorer is a common tool for anyone working in the Windows environment and sometimes being able to quickly view the content of a part of a file is all that is needed so enabling Windows Explorer Preview Pane can be quite useful.

<a href="https://msdnshared.blob.core.windows.net/media/2018/06/fileblog_04.png"><img width="200" height="142" class="alignnone wp-image-6585 size-full" alt="The preview pane is enabled from the Panes section of the toolbar" src="https://msdnshared.blob.core.windows.net/media/2018/06/fileblog_04.png" /></a>

We can see below, the contents of the csv file selected on the left are shown on the right.

<a href="https://msdnshared.blob.core.windows.net/media/2018/06/fileblog_01.png"><img width="901" height="284" class="alignnone wp-image-6565 size-full" alt="Contents of recognised file types are displayed in the preview pane" src="https://msdnshared.blob.core.windows.net/media/2018/06/fileblog_01.png" /></a>

However, if we pick a filetype that is uncommon, like the csvh file just above in the folder we don’t see anything in the preview pane.

<a href="https://msdnshared.blob.core.windows.net/media/2018/06/fileblog_02.png"><img width="796" height="285" class="alignnone wp-image-6555 size-full" alt="" src="https://msdnshared.blob.core.windows.net/media/2018/06/fileblog_02.png" /></a>

However, it is possible with a simple PowerShell cmdlet to fix this
<pre> Set-ItemProperty Registry::HKEY_CLASSES_ROOT\.csvh -Name PerceivedType -Value text</pre>
Take a look at the file now and we see the text content in the preview pane.

<a href="https://msdnshared.blob.core.windows.net/media/2018/06/fileblog_03.png"><img width="895" height="260" class="alignnone wp-image-6575 size-full" alt="when the file type is recognised the preview pane lets you read file contents" src="https://msdnshared.blob.core.windows.net/media/2018/06/fileblog_03.png" /></a>

This can save a lot of time when you want to just confirm that the latest value got written to a log file or a the details in a bcp format file (.fmt filetype) perhaps  .]]></content:encoded>
    </item>
    <item>
      <title>Keeping your important files safe and secure</title>
      <link>https://docs.microsoft.com/archive/blogs/dataplatform/keeping-your-important-files-safe-and-secure</link>
      <pubDate>Tue, 26 Jun 2018 08:37:54 GMT</pubDate>
      <dc:creator><![CDATA[Jonathan Allen msft]]></dc:creator>
      <guid
        isPermaLink="false">https://blogs.technet.microsoft.com/dataplatform/?p=6515</guid>
      <description><![CDATA[We all have important files of some sort or another but do we all look after them correctly?...]]></description>
      <content:encoded><![CDATA[We all have important files of some sort or another but do we all look after them correctly? Ensuring their security using NTFS ACLs might be the process used by many. Some will add a Read Only marker on the files that are for reference only (ISO's, backups etc) but do you do anything else to ensure the data is unchanged from the time it is written to the storage to the time that you choose to access it?

Let's see how we can use a simple PowerShell cmdlet called <code>Get-FileHash</code> to help us here.

<code>Get-FileHash</code> accepts 4 parameters:
<ul>
 	<li>Path – this is the l ocation of the file that we want to work with</li>
 	<li>Literalpath – this is used in the place of Path if we need to work with special characters (like wildcards) in our path string. No interpretation of the wildcards is done when passed in via the LiteralPath parameter</li>
 	<li>Algorithm – this is the type of hash we want to create. For our purposes we are not trying to use this for encryption purposes so MD5 and SHA1 are as valid as SHA256 or any other available options. By default SHA256 is used.</li>
 	<li>InputStream – this is used if the file is opened as a stream rather than directly from the file using the -Path parameter</li>
</ul>
We will only be using the Path parameter for the purposes of this blog but the other parameters are well explained in the Get-FileHash help content. It should be noted that the SHA1 default is a slower process than using an MD5 algorithm value. On the iso shown in the examples here the SHA256 hash regularly took 8-9s while the MD5 took 3-4s.

In its simplest form Get-FileHash returns the SHA1 hash of the file along with the filename and confirmation of the hash algorithm used
<pre>Get-FileHash -Path D:\ISO\en_sql_server_2016_service_pack_1_x64_dvd_9542248.iso | Format-List</pre>
Algorithm: SHA256
Hash: 801D596FA10E0619F63AD97413DCCC0AEAA8BD24C6894F5790547397059BBCFB
Path: D:\ISO\en_sql_server_2016_service_pack_1_x64_dvd_9542248.iso

We can pipe a result set from a previous cmdlet to Get-FileHash if we want to create hashes for a set of files
<pre>Get-ChildItem D:\HashTest | Get-FileHash</pre>
Algorithm    Hash     Path
---------    <span style="float: none;background-color: transparent;color: #333333;cursor: text;font-family: Georgia,'Times New Roman','Bitstream Charter',Times,serif;font-size: 16px;font-style: normal;font-variant: normal;font-weight: 400;letter-spacing: normal;line-height: 24px;text-align: left;text-decoration: none;text-indent: 0px">    </span> ----     ----
SHA256     A48208897580355DBFD87FF55D3AFA5153F359F1B70F9FC01770F5CEBB7BB814<span style="float: none;background-color: transparent;color: #333333;cursor: text;font-family: Georgia,'Times New Roman','Bitstream Charter',Times,serif;font-size: 16px;font-style: normal;font-variant: normal;font-weight: 400;letter-spacing: normal;line-height: 24px;text-align: left;text-decoration: none;text-indent: 0px">    </span> D:\HashTest\Database1.accdb
SHA256 <span style="float: none;background-color: transparent;color: #333333;cursor: text;font-family: Georgia,'Times New Roman','Bitstream Charter',Times,serif;font-size: 16px;font-style: normal;font-variant: normal;font-weight: 400;letter-spacing: normal;line-height: 24px;text-align: left;text-decoration: none;text-indent: 0px">    </span>6689D1E139E8AA69BCD646E2A0FC457FE1DC6EB710EBB37330AF33369EB5FE0E <span style="float: none;background-color: transparent;color: #333333;cursor: text;font-family: Georgia,'Times New Roman','Bitstream Charter',Times,serif;font-size: 16px;font-style: normal;font-variant: normal;font-weight: 400;letter-spacing: normal;line-height: 24px;text-align: left;text-decoration: none;text-indent: 0px">    </span>D:\HashTest\Document1.docx
SHA256 <span style="float: none;background-color: transparent;color: #333333;cursor: text;font-family: Georgia,'Times New Roman','Bitstream Charter',Times,serif;font-size: 16px;font-style: normal;font-variant: normal;font-weight: 400;letter-spacing: normal;line-height: 24px;text-align: left;text-decoration: none;text-indent: 0px">    </span>801D596FA10E0619F63AD97413DCCC0AEAA8BD24C6894F5790547397059BBCFB <span style="float: none;background-color: transparent;color: #333333;cursor: text;font-family: Georgia,'Times New Roman','Bitstream Charter',Times,serif;font-size: 16px;font-style: normal;font-variant: normal;font-weight: 400;letter-spacing: normal;line-height: 24px;text-align: left;text-decoration: none;text-indent: 0px">    </span>D:\HashTest\en_sql_server_2016_service_pack_1_x64_dvd_9542248.iso
SHA256 <span style="float: none;background-color: transparent;color: #333333;cursor: text;font-family: Georgia,'Times New Roman','Bitstream Charter',Times,serif;font-size: 16px;font-style: normal;font-variant: normal;font-weight: 400;letter-spacing: normal;line-height: 24px;text-align: left;text-decoration: none;text-indent: 0px">    </span>63EA7EC057B089A0CFCBA568B40FC5C5016A33865BE77D3822321B369142E60A <span style="float: none;background-color: transparent;color: #333333;cursor: text;font-family: Georgia,'Times New Roman','Bitstream Charter',Times,serif;font-size: 16px;font-style: normal;font-variant: normal;font-weight: 400;letter-spacing: normal;line-height: 24px;text-align: left;text-decoration: none;text-indent: 0px">    </span>D:\HashTest\Hash_Database1_accdb.csvh
SHA256 <span style="float: none;background-color: transparent;color: #333333;cursor: text;font-family: Georgia,'Times New Roman','Bitstream Charter',Times,serif;font-size: 16px;font-style: normal;font-variant: normal;font-weight: 400;letter-spacing: normal;line-height: 24px;text-align: left;text-decoration: none;text-indent: 0px">    </span>B403128B954520D356A9BD0C0B474FF9D09B08BD3494C6BE091ECC1107390067 <span style="float: none;background-color: transparent;color: #333333;cursor: text;font-family: Georgia,'Times New Roman','Bitstream Charter',Times,serif;font-size: 16px;font-style: normal;font-variant: normal;font-weight: 400;letter-spacing: normal;line-height: 24px;text-align: left;text-decoration: none;text-indent: 0px">    </span>D:\HashTest\Hash_Document1_docx.csvh
SHA256 <span style="float: none;background-color: transparent;color: #333333;cursor: text;font-family: Georgia,'Times New Roman','Bitstream Charter',Times,serif;font-size: 16px;font-style: normal;font-variant: normal;font-weight: 400;letter-spacing: normal;line-height: 24px;text-align: left;text-decoration: none;text-indent: 0px">    </span>ECA46DBC21EC3D18CC6065A7D33547E296BFAAEAFE9E587FCD142D2391D8CD91 <span style="float: none;background-color: transparent;color: #333333;cursor: text;font-family: Georgia,'Times New Roman','Bitstream Charter',Times,serif;font-size: 16px;font-style: normal;font-variant: normal;font-weight: 400;letter-spacing: normal;line-height: 24px;text-align: left;text-decoration: none;text-indent: 0px">    </span>D:\HashTest\Hash_Presentation1_pptx.csvh
SHA256 <span style="float: none;background-color: transparent;color: #333333;cursor: text;font-family: Georgia,'Times New Roman','Bitstream Charter',Times,serif;font-size: 16px;font-style: normal;font-variant: normal;font-weight: 400;letter-spacing: normal;line-height: 24px;text-align: left;text-decoration: none;text-indent: 0px">    </span>4861C11FEC24C3CDE663641AF92C2069EFA10F51A79E24AB92F7B1F684B7CEC0 <span style="float: none;background-color: transparent;color: #333333;cursor: text;font-family: Georgia,'Times New Roman','Bitstream Charter',Times,serif;font-size: 16px;font-style: normal;font-variant: normal;font-weight: 400;letter-spacing: normal;line-height: 24px;text-align: left;text-decoration: none;text-indent: 0px">    </span>D:\HashTest\ISOArchive.csv
SHA256 <span style="float: none;background-color: transparent;color: #333333;cursor: text;font-family: Georgia,'Times New Roman','Bitstream Charter',Times,serif;font-size: 16px;font-style: normal;font-variant: normal;font-weight: 400;letter-spacing: normal;line-height: 24px;text-align: left;text-decoration: none;text-indent: 0px">    </span>B9100952467E7403F19CF450B150447B4D1B1107FB1D1A6A6D09446EEE6CB778 <span style="float: none;background-color: transparent;color: #333333;cursor: text;font-family: Georgia,'Times New Roman','Bitstream Charter',Times,serif;font-size: 16px;font-style: normal;font-variant: normal;font-weight: 400;letter-spacing: normal;line-height: 24px;text-align: left;text-decoration: none;text-indent: 0px">    </span>D:\HashTest\Presentation1.pptx

The output from the cmdlet is an object so we can reference the results from the above by setting the output to a variable and referencing the array index for the value we need.
<pre>$IsoHash = Get-ChildItem D:\HashTest | Get-FileHash -Algorithm MD5
$IsoHash[2].Hash
33B324D311FDFC68A9A3E3405B8CD222
$IsoHash[2].Path
D:\ISO\en_sql_server_2016_service_pack_1_x64_dvd_9542248.iso
$IsoHash[2].Algorithm
MD5
</pre>
We can also use the whole object with the <em><strong>-match</strong></em> regex comparison to search for files that we have already calculated a hash.
<pre>$FileToConfirm = "en_sql_server_2016"
$r = $IsoHash.path -match $FileToConfirm
if ($r) {
"$r found" | Write-Output
}
else {
"No details found for $FileToConfirm" | Write-Output
}
D:\HashTest\en_sql_server_2016_service_pack_1_x64_dvd_9542248.iso found</pre>
Let’s now work on placing all the Get-FileHash output into a file so that we can reference it in months to come to check our iso's etc are guaranteed to be safe to use.
<pre># export to a file
$IsoHash | Export-csv -Path "D:\HashTest\ISOArchive.csvh"</pre>
Opening this file in Notepad we see the information that we need, safely stored for us.

#TYPE Microsoft.Powershell.Utility.FileHash
"Algorithm","Hash","Path"
"MD5","27C2D3E6786CE78F77F45F8B6AEEC97C","D:\HashTest\Database1.accdb"
"MD5","4C3661354D2C104963A8F80749818D6A","D:\HashTest\Document1.docx"
"MD5","33B324D311FDFC68A9A3E3405B8CD222","D:\HashTest\en_sql_server_2016_service_pack_1_x64_dvd_9542248.iso"
"MD5","2141E775DBC0328775CDFED0E7B28058","D:\HashTest\Hash_Database1_accdb.csvh"
"MD5","8DC877E71CE2575B8E34C933D9D2384F","D:\HashTest\Hash_Document1_docx.csvh"
"MD5","BADBCBF3D8ACB5B705A32A9697EBB8A4","D:\HashTest\Hash_Presentation1_pptx.csvh"
"MD5","488890AE63C647350CCCC9E3C47C3D8D","D:\HashTest\ISOArchive.csv"
"MD5","AF6233A16AAEBFD23373EA987BA0D497","D:\HashTest\Presentation1.pptx"

Now, imagine many months pass and we want to use an iso to install some software or we want to ensure that script content hasn’t changed since it was placed into the store location. We are able to generate a hash of the content just before we use it and compare that hash to the value generated when the store was created. Any changes to the file contents will result in a different hash value and we will therefore know that the content has changed and should not be trusted.
<pre># read hash from file and compare with newly calculated hash
$ISOArchive = @()
$ISOArchive = Import-csv -path "D:\hashtest\ISOArchive.csvh"

# choose the file we want to verify and get the hash of it right now
$FileToCheck = "D:\hashtest\en_sql_server_2016_service_pack_1_x64_dvd_9542248.iso"
$NewFileHash = Get-FileHash $FileToCheck -Algorithm MD5

# check if the new hash is found in the archive
$match = ($ISOArchive.hash) -match [regex]::Escape($NewFileHash.Hash)
if ($match) {
    "The hash for $($match.path) matches the archive value. It is safe to use." | Write-Output
}
else { # check if the file is found in the archive
    $exists = (($ISOArchive.path) -match [regex]::Escape($NewFileHash.Path)) -and (($ISOArchive.algorithm) -match [regex]::Escape($NewFileHash.Algorithm))
    if ($exists) { # if the file exists and the hash wasn’t found then the file has changed
        "The hash for $($match.path) does not match the archive value. It is NOT safe." | Write-Warning
    }
    else { # if the file isn’t found then we don’t have a hash that we can compare with
        if (!((($ISOArchive.path) -match [regex]::Escape($NewFileHash.Path)))) {
            "It doesnt look like there is a hash value for $($Newfilehash.path) in the archive." | Write-Output
        }
        else { # we have to compare hashes generated with the same algorithm
            "Please re-test $($newfilehash.path) using the correct hash algorithm" | Write-Output
        }
    }
}

</pre>
I hope you can see how useful this is to ensure confidence in important files that you place on your network.]]></content:encoded>
    </item>
    <item>
      <title>Somewhere to put that data while your script runs</title>
      <link>https://docs.microsoft.com/archive/blogs/dataplatform/somewhere-to-put-that-data-while-your-script-runs</link>
      <pubDate>Mon, 21 May 2018 08:54:37 GMT</pubDate>
      <dc:creator><![CDATA[Jonathan Allen msft]]></dc:creator>
      <guid
        isPermaLink="false">https://blogs.technet.microsoft.com/dataplatform/?p=6505</guid>
      <description><![CDATA[Ever writing a script and want to quickly place some data in a file while in the middle of your...]]></description>
      <content:encoded><![CDATA[Ever writing a script and want to quickly place some data in a file while in the middle of your process? Try out <strong>[System.Io.Path]::GetTempFileName()</strong>. This method gives you a file with a random, unused file name in your local temp folder where you can write out interim information without worrying about testing whether the file exists or what to call the file etc.

Let's take a closer look...

Open up your favourite PowerShell script editor and run <pre>[System.Io.Path]::GetTempFileName()</pre>
You will end up with output something like this
<pre>C:\Users\username\AppData\Local\Temp\tmpE412.tmp</pre>
Check in Windows Explorer - the file has actually been created so what you do next might need some consideration. If you are wanting to write data out to the file then if your output process accepts an Append style functionality then you need to do that,
<pre>$F = [System.Io.Path]::GetTempFileName() 

Get-Process | Out-File $F -Append

"Process information stored in $F" | Write-Output</pre>
 otherwise you might need to actually delete this file and then use your data output process to create it.
<pre>
$F = [System.Io.Path]::GetTempFileName() 
$F | Remove-Item -Force

Get-Process | Out-File $F

"Process information stored in $F" | Write-Output</pre>

I encourage you to review all the other useful methods of the [System.Io.Path] class <a href="https://msdn.microsoft.com/en-us/library/system.io.path_methods(v=vs.110).aspx" target="_blank">here on MSDN</a>.


]]></content:encoded>
    </item>
    <item>
      <title>How to find out what you need to know</title>
      <link>https://docs.microsoft.com/archive/blogs/dataplatform/how-to-find-out-what-you-need-to-know</link>
      <pubDate>Mon, 07 May 2018 08:45:28 GMT</pubDate>
      <dc:creator><![CDATA[Jonathan Allen msft]]></dc:creator>
      <guid
        isPermaLink="false">https://blogs.technet.microsoft.com/dataplatform/?p=6495</guid>
      <description><![CDATA[For as long as I can remember I have come across problems where I need to know the range of valid...]]></description>
      <content:encoded><![CDATA[For as long as I can remember I have come across problems where I need to know the range of valid options for a setting so that I can make an appropriate change, but I haven’t know the right values to use.

For  example - If you wanted  to change the powerplan of your computer you might know that there is a 'High Performance' setting. Perhaps you know there is a 'Balanced' option – it’s the default on all Windows OS since 2008 but, would you guess that there is a 'Low Performance' or 'Power saver' or 'battery saver.

How about SQL Server Login Mode options? We talk about SQL Server Authentication and Mixed Mode but what are the actual values to use when you want to change them programmatically and what other values are there?

Until recently I used to start searching online and perhaps encounter a blog where someone has done the investigation already or I would find a MSDN or TechNet article that explain it all.

However, you can do this all from the safety of your own script editor. Let's see how.

<strong>Example 1</strong> – Getting the valid values for the SMO LoginMode property of a SQL Server.

Step 1 – create an SMO object that represents the server we  are interested in and then set a different variable to  be the LoginMode

<pre>$SMOServer = New-Object ('Microsoft.SqlServer.Management.Smo.Server') "$env:computername\sql2016"

$LMode = $SMOServer.LoginMode</pre>


So, $LMode is now representing the LoginMode for our server, we can now see the variable type by running

<pre>$LMode.GetType()</pre>


For me this returns

<pre>IsPublic IsSerial Name                                     BaseType
-------- -------- ----                                     --------
True     True     ServerLoginMode                          System.Enum
</pre>

Step 2 - We can see in last output of Step 1 that the Name of the variable type is ServerLoginMode. We know that we are using the SMO object Microsoft.SqlServer.Management.Smo.Server (it’s the object type we used when we created $SMOServer) so we can now get all the valid values for LoginMode by executing <strong>[System.Enum]::GetValues</strong>.

<pre>[System.Enum]::GetValues([Microsoft.SqlServer.Management.SMO.ServerLoginMode])</pre>

The results of this are:

<pre>Normal 
Integrated 
Mixed 
Unknown</pre>

Which is exactly what we were looking for.

<strong>Example 2</strong> – getting the valid values for a Windows Service status

Step 1 – In PowerShell the most common interaction with a Windows Service is via the Get-Service cmdlet. Running this gives us mostly services that are Running or Stopped but perhaps we want to build a pester test that is checking that a service is running or at least starting up. How can we check for the right values?

Working once again with a SQL Server service we run

<pre>Get-Service | ? Name -match mssql\$</pre>

And we get a list of SQL Server database services that are present on the local computer

<pre>Status   Name               DisplayName
------   ----               -----------
Stopped  MSSQL$SQL2008A     SQL Server (SQL2008A)
Stopped  MSSQL$SQL2008B     SQL Server (SQL2008B)
Running  MSSQL$SQL2014      SQL Server (SQL2014)
Running  MSSQL$SQL2016      SQL Server (SQL2016)</pre>

Step 2 – as in example 1 we need to create a variable that represents the service so that we can investigate it so we don’t need all 4 services getting in the way from the above so lets alter the code a little

<pre>$svc = Get-Service | ? Name -match mssql\$ | select -First 1</pre>

Step 3 – we now have $svc as a single service, lets set another variable to the service status

<pre>$svc_status = $svc.Status</pre>

And then we just check its type as before
<pre>$svc_status.GetType()</pre>

<pre>IsPublic IsSerial Name                                     BaseType
-------- -------- ----                                     --------
True     True     ServiceControllerStatus                  System.Enum</pre>

Ok, this makes sense, we have a Name that is ServiceControllerStatus so lets pump this through <strong>[System.Enum]::GetValues</strong> …

<pre>Unable to find type [ServiceControllerStatus].</pre>

Oops. I have trimmed the error a bit, but we get an error, not a nice message about the values we can use. We need to get more information about the $svc_status variable...

To see all the information we need to get past the defaultpropertyset of the GetType() method, we do this with

<pre>$svc_status.GetType() | select *</pre>

The resulting list is huge and I'll leave the review of all those details as a step for the readers that are interested. In there however, there are a couple of property names that look relevant to our investigations – <strong>NameSpace</strong> and <strong>FullName</strong> in particular look promising. Let's get the values that they hold

<pre>$svc_status.GetType() | select fullname, namespace</pre>

This gives us:
<pre>
FullName                                      Namespace
--------                                      ---------
System.ServiceProcess.ServiceControllerStatus System.ServiceProcess</pre>

Step 4
We've got what we need, let's make the final attempt at getting the information we came here for

<pre>[System.Enum]::GetValues([system.ServiceProcess.ServiceControllerStatus])</pre>

And it indeed gives us the list of valid Service Status values that we were after:

<pre>Stopped 
StartPending
StopPending
Running
ContinuePending
PausePending
Paused</pre>

Hopefully this has shown you the technique needed to investigate variable and object types using <strong>.GetType()</Strong> and <strong>[System.Enum]::GetValues</Strong> and how to locate valid values to be used in your code. Please let us know in the comments if this is useful to you or if you have any questions.]]></content:encoded>
    </item>
    <item>
      <title>Fill Factor and Data Compression</title>
      <link>https://docs.microsoft.com/archive/blogs/dataplatform/fill-factor-and-data-compression</link>
      <pubDate>Mon, 30 Apr 2018 08:00:22 GMT</pubDate>
      <dc:creator><![CDATA[Chris Lound]]></dc:creator>
      <guid
        isPermaLink="false">https://blogs.technet.microsoft.com/dataplatform/?p=6345</guid>
      <description><![CDATA[Another post by our excellent US PFE Susan Van Eyck goes though Fill Factor and Data...]]></description>
      <content:encoded><![CDATA[Another post by our excellent US PFE Susan Van Eyck goes though Fill Factor and Data Compression!

&nbsp;

I got a pair of interesting questions from a colleague this week:
<ol>
 	<li>If you’re using fill factor then enable data compression, is the fill factor still enforced?</li>
 	<li>If you aren’t using fill factor and enable data compression does it <u>create</u> free space on the original pages?</li>
</ol>
The answer to #2 is “No” because to enable compression (row or page) you rebuild an index, and that involves writing the data out to a new set of data (or index) pages then dropping the original index.  Note the page numbers in the illustration below (bonus points if you can explain the funky numbering in the original index).  Since this index was rebuilt <u>without</u> a fill factor, each page was filled to capacity.  Only the last page might have room for more data (represented here by the lighter colors):

<a href="https://msdnshared.blob.core.windows.net/media/2018/04/image91.png"><img width="683" height="392" title="image" alt="image" src="https://msdnshared.blob.core.windows.net/media/2018/04/image_thumb90.png" border="0" /></a>

&nbsp;

As for question #1, I did a bit of research and found a helpful <a href="https://blogs.msdn.microsoft.com/sqlserverstorageengine/2011/05/12/data-compression-and-fill-factor/">blog post</a> from Sunil Agarwal (one of the SQL Server Program Managers) confirming that the 2 features are compatible.  I wrote a demo script based on Sunil’s, and thought it would be interesting to share with you – along with a few notes about data compression.

To start, we’ll create a table in tempdb then fill it with a lot of repetitive data:

USE tempdb;

CREATE TABLE dbo.BogusData (

BogusDataId INT IDENTITY,

WideColumn CHAR(100) DEFAULT REPLICATE( 'A', 50 )

);

INSERT INTO dbo.BogusData ( WideColumn ) DEFAULT VALUES;

GO 10000

That CHAR(100) column is a good target for <b>Row Compression</b> since we’re using only half the reserved space with strings 50 characters in length.  You can think of Row Compression as smooshing out leftover space from columns.  Here are some examples of where it will be useful:

<a href="https://msdnshared.blob.core.windows.net/media/2018/04/image92.png"><img width="686" height="325" title="image" alt="image" src="https://msdnshared.blob.core.windows.net/media/2018/04/image_thumb91.png" border="0" /></a>

&nbsp;

&nbsp;

And all that repetitive data is a good target for <b>Page Compression</b> which stores 1 copy of a repeated column values in the page header then substitutes a smaller placeholder in the body of the page (dictionary compression).  My index is bit wonky, and the compression process is a bit more involved, but you get the idea – a lot more wombats per page read.

<a href="https://msdnshared.blob.core.windows.net/media/2018/04/image93.png"><img width="532" height="294" title="image" alt="image" src="https://msdnshared.blob.core.windows.net/media/2018/04/image_thumb92.png" border="0" /></a>

&nbsp;

The impact of data compression will depend greatly on both your data types and your data.  For instance, a CHAR(10) column that always contains 10 characters won’t benefit from row-compression.  Neither will a VARCHAR(n) column doesn’t have extra space to eliminate.  A unique index on a column like SocialSecurityNumber won’t benefit from either row or page compression (assuming you’ve chosen the data type and width wisely).  And repetitive data can only page-compressed if the repeated values occur on the same data page.  Imagine an index on EmployeeId that includes LastName.  If there are 2000 Smiths are scattered across 10,000 data pages, dictionary compression won’t be of much use.  Onto our demonstration!

In the demos below we’ll start by adding a clustered index to our table, then we’ll look at the impact of adding a fill factor, row compression and finally page compression (which includes row compression).  We’ll check the index’s size and space usage after each step (although the query to get those values is only printed once).  Let’s see what happens to those values across our 4 test cases.
<ol>
 	<li>Add a Clustered Index</li>
</ol>
CREATE CLUSTERED INDEX ci_BogusDataId ON dbo.BogusData ( BogusDataId );

SELECT page_count, avg_page_space_used_in_percent

FROM sys.dm_db_index_physical_stats( DB_ID(), OBJECT_ID( 'dbo.BogusData' ), NULL, NULL, 'DETAILED' )

WHERE index_level = 0;

<a href="https://msdnshared.blob.core.windows.net/media/2018/04/image94.png"><img width="464" height="84" title="image" alt="image" src="https://msdnshared.blob.core.windows.net/media/2018/04/image_thumb93.png" border="0" /></a>

&nbsp;

Note that our data pages are packed full.  This helps keep the cost of I/O down since we get more data rows per page read.

[<i>Note:  We’re using <b>DETAILED</b> mode in the query to return the avg_page_space_used_in_percent value, but this requires that <u>all</u> the index’s data pages be read.  Be <u>very</u> cautious using this mode in a production  environment.</i>]
<ol start="2">
 	<li>Add a Fill Factor</li>
</ol>
ALTER INDEX ci_BogusDataId ON dbo.BogusData REBUILD WITH ( FILLFACTOR = 80 );

<a href="https://msdnshared.blob.core.windows.net/media/2018/04/image95.png"><img width="476" height="83" title="image" alt="image" src="https://msdnshared.blob.core.windows.net/media/2018/04/image_thumb94.png" border="0" /></a>

&nbsp;

We’ve now got about 20% free space on each data page – good for delaying page splits, but we’ve also made the index 20% larger which will negatively impact queries involving index scans as there are now fewer row per page.  Keep this trade-off in mind when implementing fill factors.
<ol start="3">
 	<li>Add Row Compression (the specified fill factor will persist through the rebuild)</li>
</ol>
ALTER INDEX ci_BogusDataId ON dbo.BogusData REBUILD WITH ( DATA_COMPRESSION = ROW );

<a href="https://msdnshared.blob.core.windows.net/media/2018/04/image96.png"><img width="482" height="76" title="image" alt="image" src="https://msdnshared.blob.core.windows.net/media/2018/04/image_thumb95.png" border="0" /></a>

&nbsp;

This is a move in the right direction!  We’ve still got free space for new data, and “smooshing” out the empty spaces has dropped the index size by more than 40%.  We might have expected a 50% drop since we’re only using half of the column’s 100 character width, but some page space is now is being used to store information about the uncompressed state of the column.
<ol start="4">
 	<li>Add Page Compression (includes Row Compression which will be applied first)</li>
</ol>
ALTER INDEX ci_BogusDataId ON dbo.BogusData REBUILD WITH ( DATA_COMPRESSION = PAGE );

<a href="https://msdnshared.blob.core.windows.net/media/2018/04/image97.png"><img width="494" height="88" title="image" alt="image" src="https://msdnshared.blob.core.windows.net/media/2018/04/image_thumb96.png" border="0" /></a>

&nbsp;

Once again, the fill factor persists, and due to the very bogus, super-repetitive nature of our data the index is down to a mere 18 pages!  Note that you’re unlikely to see such amazing gains with real-world data.

Remember to drop the demo table when you’re done:

DROP TABLE dbo.BogusData;

If you’re thinking about using either fill factor or data compression, here’s some additional reading that covers some of the factors you’ll want to take into consideration:

<a href="https://docs.microsoft.com/en-us/sql/relational-databases/indexes/specify-fill-factor-for-an-index">Specify Fill Factor for an Index</a>

<a href="https://docs.microsoft.com/en-us/sql/relational-databases/data-compression/data-compression">Data Compression</a>

And lastly,  I’ve attached a script that has the full set of demo scripts to save you a little copy/paste effort!

<a href="https://msdnshared.blob.core.windows.net/media/2018/04/20180404_FillFactorAndDataCompression.zip">20180404_FillFactorAndDataCompression</a>

&nbsp;

Happy Exploring!

&nbsp;

&nbsp;]]></content:encoded>
    </item>
    <item>
      <title>Microseconds Or Milliseconds</title>
      <link>https://docs.microsoft.com/archive/blogs/dataplatform/microseconds-or-milliseconds</link>
      <pubDate>Mon, 23 Apr 2018 08:00:08 GMT</pubDate>
      <dc:creator><![CDATA[Oren Bouni]]></dc:creator>
      <guid
        isPermaLink="false">https://blogs.technet.microsoft.com/dataplatform/?p=6185</guid>
      <description><![CDATA[&nbsp;

While working with a customer on creating some Extended Event to collect performance...]]></description>
      <content:encoded><![CDATA[&nbsp;

While working with a customer on creating some Extended Event to collect performance trace, I was if asked if the duration is in microseconds or milliseconds.

Thinking to myself, "well in SQL 2005 and later it is in microseconds", although the Profiler UI has an option to display the value in either milliseconds or microseconds.  So surly in Xevent it will stay the same?

It was interesting, and I started to look for more information on that.

Well apparently, it depends, it can be in microseconds or milliseconds second depending on the event.

Doing some more reading on that subject shown that with the evolution of computing the processor speed as changed and by that also the precision that was expected from the counter has change to milliseconds sometimes.

Looking at these events, <b><u>Wait_info</u></b>, <b><u>wait_info_external</u>, </b>this events are calculated in microseconds and <b><u>task_completed</u></b> is in milliseconds.

looking into this three system tables <b>sys.dm_xe_objects</b>, <b>sys.dm_xe_packages</b> and <b>sys.dm_xe_object_columns</b>, shows us all the needed information and if it is in Micro or Milli seconds.

The information of the duration time is taken from <b>sys.dm_xe_object_columns </b>under the column description<b>, </b>there we will see<b> </b>the duration time in microseconds or milliseconds.

The duration column will be shown with a description:

“<b><i>Wait duration in milliseconds</i></b>” or “<b><i>Indicates the total time (in microseconds) that has taken the...</i></b><i>” </i>

To extract the unit, I have added a substring and charindex that looks for the Milli or Micro in the description columns.

Here is the simple but useful (well at least for me) script.

&nbsp;

[sql]Select p.name package, o.name event,
Case
   When c.description Like '%milli%' Then Substring(c.description, Charindex('milli', c.description),12)
   When c.description like '%micro%' Then Substring(c.description, Charindex('micro', c.description),12)
   Else 'Null'
End Unit, o.description, p.description
From sys.dm_xe_objects o Join sys.dm_xe_packages p On o.package_guid = p.guid
Join sys.dm_xe_object_columns c On o.name = c.object_name
Where o.object_type = 'event' and c.name ='duration'[/sql]

&nbsp;

<a href="https://msdnshared.blob.core.windows.net/media/2018/04/clip_image0023.jpg"><img width="1038" height="375" title="clip_image002" alt="clip_image002" src="https://msdnshared.blob.core.windows.net/media/2018/04/clip_image002_thumb4.jpg" border="0" /></a><b></b>

<b>Oren Bouni</b>]]></content:encoded>
    </item>
    <item>
      <title>You really need to look out for msdb security</title>
      <link>https://docs.microsoft.com/archive/blogs/dataplatform/you-really-need-to-look-out-for-msdb-security</link>
      <pubDate>Mon, 16 Apr 2018 08:00:20 GMT</pubDate>
      <dc:creator><![CDATA[SQL by Edinson Medina]]></dc:creator>
      <guid
        isPermaLink="false">https://blogs.technet.microsoft.com/dataplatform/?p=6085</guid>
      <description><![CDATA[So msdb, we don't really think about it that much right? We know it holds the metadata for our Jobs...]]></description>
      <content:encoded><![CDATA[So msdb, we don't really think about it that much right? We know it holds the metadata for our Jobs and Schedules, also some good info on page corruptions, as well as backups, and much more.

But how often do we think about it’s security? You may already know that the Guest user is enabled for msdb, but you should also know that this should <strong>not</strong> be disabled, as pointed here <a href="https://support.microsoft.com/en-gb/help/2539091/you-should-not-disable-the-guest-user-in-the-msdb-database-in-sql-serv" title="https://support.microsoft.com/en-gb/help/2539091/you-should-not-disable-the-guest-user-in-the-msdb-database-in-sql-serv">https://support.microsoft.com/en-gb/help/2539091/you-should-not-disable-the-guest-user-in-the-msdb-database-in-sql-serv</a>

Another configuration that is ON for msdb by default is the Trustworthy setting, and again is a setting that should <strong>not</strong> be turn off, some details on what this setting is <a href="https://support.microsoft.com/en-us/help/2183687/guidelines-for-using-the-trustworthy-database-setting-in-sql-server" title="https://support.microsoft.com/en-us/help/2183687/guidelines-for-using-the-trustworthy-database-setting-in-sql-server">https://support.microsoft.com/en-us/help/2183687/guidelines-for-using-the-trustworthy-database-setting-in-sql-server</a>

but what does this mean, and could it be a problem? The short answer is YES - let's check this out.

Let's start with the fact that a lot of times our practice is to make <strong>sa</strong> the owner of all databases, a practice that although providing easier management, does not do us any favours from the security perspective.

So imagine the scenario that we have a third party app that does some automation and scheduling tasks for us, and we have been asked to add the app login to the db_owner role in msdb

So we create the login, and map it to msdb db_owner, also we make sure its not a sysadmin.

<a href="https://msdnshared.blob.core.windows.net/media/2018/04/image39.png"><img width="413" height="374" title="image" alt="image" src="https://msdnshared.blob.core.windows.net/media/2018/04/image_thumb39.png" border="0" /></a>     <a href="https://msdnshared.blob.core.windows.net/media/2018/04/image40.png"><img width="415" height="376" title="image" alt="image" src="https://msdnshared.blob.core.windows.net/media/2018/04/image_thumb40.png" border="0" /></a>

Let's just check we can now connect to SQL with the login.

<a href="https://msdnshared.blob.core.windows.net/media/2018/04/image36.png"><img width="244" height="149" title="image" alt="image" src="https://msdnshared.blob.core.windows.net/media/2018/04/image_thumb36.png" border="0" /></a>

Now, the fact that this login is a db_owner (this also apply if the impersonate permission is given to the user) on msdb allows it to exercise the <strong>impersonation privilege</strong>, and  we could then impersonate the owner of the database, which is? You guessed it, the sa account (by the way the sa account could be disabled but this just prohibits connecting with the sa credentials, but we connected to SQL with applogin, that's why the permissions are still working fine).

<a href="https://msdnshared.blob.core.windows.net/media/2018/04/image37.png"><img width="244" height="167" title="image" alt="image" src="https://msdnshared.blob.core.windows.net/media/2018/04/image_thumb37.png" border="0" /></a>

From here everything gets quite simple, getting data from another databases, or create another user, or make it part if the sysadmin server role even though we logged in to the server with the applogin credentials.

<a href="https://msdnshared.blob.core.windows.net/media/2018/04/image41.png"><img width="638" height="369" title="image" alt="image" src="https://msdnshared.blob.core.windows.net/media/2018/04/image_thumb41.png" border="0" /></a>

So what should we do then?  We cannot disable the Trustworthy setting, but we can manage the owner of the database and more importantly the permissions we give to applogin. Instead of providing <strong>db_owner</strong> we should look the necessary permissions for it, and <em>fit it on the least privilege model</em>, check if you can use one of the existing db roles in msdb (<a href="https://docs.microsoft.com/en-us/sql/relational-databases/security/authentication-access/database-level-roles#msdb-roles" title="https://docs.microsoft.com/en-us/sql/relational-databases/security/authentication-access/database-level-roles#msdb-roles">https://docs.microsoft.com/en-us/sql/relational-databases/security/authentication-access/database-level-roles#msdb-roles</a>) if not, them assign the specific needed permissions to applogin.

Hope this helps you on maintaining your data secured.]]></content:encoded>
    </item>
    <item>
      <title>Troubleshooting Using Ring Buffers</title>
      <link>https://docs.microsoft.com/archive/blogs/dataplatform/troubleshooting-using-ring-buffers</link>
      <pubDate>Mon, 09 Apr 2018 08:00:38 GMT</pubDate>
      <dc:creator><![CDATA[Dr. David Thulborn]]></dc:creator>
      <guid
        isPermaLink="false">https://blogs.technet.microsoft.com/dataplatform/?p=5875</guid>
      <description><![CDATA[Working at a customer site, and I happened to notice that the Exception ring buffer contained a...]]></description>
      <content:encoded><![CDATA[<p><br></p>
<p>Working at a customer site, and I happened to notice that the Exception ring buffer contained a lot (and I mean A.L.O.T.) of errors. Ring Buffers were introduced in SQL 2008, and contain a fixed size cycle of records for the ring buffer they pertain to. Now… note – sys.dm_os_ring_buffers is a DMV which is NOT SUPPORTED. and future compatibility is NOT guaranteed… But… that doesn’t mean we can’t use them – right <img class="wlEmoticon wlEmoticon-winkingsmile" alt="Winking smile" src="https://msdnshared.blob.core.windows.net/media/2018/03/wlEmoticon-winkingsmile.png">…</p>
<p>So the ring buffers available are :-</p>
<p>[sql]SELECT DISTINCT ring_buffer_type FROM sys.dm_os_ring_buffers[/sql]</p>
<p>Which gives me:</p>
<p><a href="https://msdnshared.blob.core.windows.net/media/2018/03/image358.png"><img width="280" height="233" title="image" alt="image" src="https://msdnshared.blob.core.windows.net/media/2018/03/image_thumb332.png" border="0"></a></p>
<p>For this Blog Post, I am particularly interested in the RING_BUFFER_EXCEPTION ring buffer, as this will show me the exceptions which have been occurring across the instance. Since these exceptions might only be a severity of 15 or below, it is entirely possible that we don’t know they are occurring – unless we look in the ring buffer.</p>
<p>But how?</p>
<p>If I cast the ring buffer record as an XML datatype, I can see the format of the payload:</p>
<p><a href="https://msdnshared.blob.core.windows.net/media/2018/03/image359.png"><img width="381" height="166" title="image" alt="image" src="https://msdnshared.blob.core.windows.net/media/2018/03/image_thumb333.png" border="0"></a></p>
<p>So, I really need to break this XML down so I can do something useful with it, and whilst I am at it, I can use the sys.messages table to turn the Error (&lt;Error&gt;926&lt;/Error&gt; in this case) into something slightly more meaningful, with:</p>
<p>[sql]SELECT&lt;br&gt;
      DATEADD(ms, -1 * (info.ms_ticks - XEvent.value('(@time)[1]', 'INT')), GETDATE()) AS [TimeStamp]&lt;br&gt;
    , XEvent.value('(@id)[1]', 'INT') AS [ID]&lt;br&gt;
    , XEvent.value('(./Exception/Error)[1]', 'INT') AS [Error]&lt;br&gt;
    , XEvent.value('(./Exception/Severity)[1]', 'INT') AS [Severity]&lt;br&gt;
    , XEvent.value('(./Exception/State)[1]', 'INT') AS [State]&lt;br&gt;
    , XEvent.value('(./Exception/UserDefined)[1]', 'INT') AS [UserDefined]&lt;br&gt;
    , sm.text&lt;br&gt;
FROM (&lt;br&gt;
       SELECT&lt;br&gt;
           CAST(record AS XML) AS [target_data]&lt;br&gt;
       FROM&lt;br&gt;
           sys.dm_os_ring_buffers&lt;br&gt;
       WHERE&lt;br&gt;
           ring_buffer_type = 'RING_BUFFER_EXCEPTION' ) as XEventsRingBuffer&lt;br&gt;
CROSS APPLY target_Data.nodes('/Record') AS XEventData (XEvent)&lt;br&gt;
INNER JOIN sys.messages sm&lt;br&gt;
ON sm.message_id = XEvent.value('(./Exception/Error)[1]', 'INT')&lt;br&gt;
CROSS APPLY sys.dm_os_sys_info info[/sql]</p>
<p>Resulting in:</p>
<p><a href="https://msdnshared.blob.core.windows.net/media/2018/03/image362.png"><img width="810" height="165" title="image" alt="image" src="https://msdnshared.blob.core.windows.net/media/2018/03/image_thumb336.png" border="0"></a></p>
<p>Ok – that’s cool. So I can now use the ring buffer to extract the XML of the exception and turn it into something a bit more human readable.</p>
<p>What if I need to investigate further? What if the Exception leads me to something not so straight-forward? Well, XEvents to the rescue:</p>
<p>[sql]-- If the Event Sessions already Exists, drop it&lt;br&gt;
IF EXISTS ( SELECT 1 FROM [sys].[server_event_sessions] WHERE [name] = N'XE_Error_Capture')&lt;br&gt;
BEGIN&lt;br&gt;
    DROP EVENT SESSION [XE_Error_Capture] ON SERVER;&lt;br&gt;
END&lt;br&gt;
-- Create the XEvent to catch the error&lt;br&gt;
CREATE EVENT SESSION [XE_Error_Capture]&lt;br&gt;
ON SERVER&lt;br&gt;
ADD EVENT [sqlserver].[error_reported] (&lt;br&gt;
    ACTION([sqlserver].[session_id],&lt;br&gt;
           [sqlserver].[database_name],&lt;br&gt;
           [sqlserver].[tsql_stack])&lt;br&gt;
WHERE ([error_number]=123456789)) -- Replace with Error Number&amp;nbsp;&amp;nbsp; &lt;br&gt;
ADD TARGET [package0].[event_file](SET FILENAME = N'C:\Temp\XE_Error_Capture.xel')&lt;br&gt;
GO&lt;br&gt;
-- Start the Session&lt;br&gt;
ALTER EVENT SESSION [XE_Error_Capture] ON SERVER&lt;br&gt;
STATE = START;&lt;br&gt;
GO[/sql]</p>
<p><br></p>
<p>I can now use an XEvent to specifically hunt out the Errors previously discovered in the ring buffer – including the database which they are occurring in, and the tsql_stack (should I need it). This can be broken down in the usual manner, or even viewed directly with the XEvent Viewer built right into SSMS.</p>
<p>Neat <img class="wlEmoticon wlEmoticon-smile" alt="Smile" src="https://msdnshared.blob.core.windows.net/media/2018/03/wlEmoticon-smile3.png"></p>]]></content:encoded>
    </item>
    <item>
      <title>Advantages and Annoyances of Filtered Indexes</title>
      <link>https://docs.microsoft.com/archive/blogs/dataplatform/advantages-and-annoyances-of-filtered-indexes</link>
      <pubDate>Mon, 02 Apr 2018 08:00:49 GMT</pubDate>
      <dc:creator><![CDATA[Chris Lound]]></dc:creator>
      <guid
        isPermaLink="false">https://blogs.technet.microsoft.com/dataplatform/?p=5635</guid>
      <description><![CDATA[This weeks great post by our US Senior PFE Susan Van Eyck talks about the wonders of filtered...]]></description>
      <content:encoded><![CDATA[This weeks great post by our US Senior PFE <strong>Susan Van Eyck</strong> talks about the wonders of filtered indexes!

Check it out!

&nbsp;

Today’s Byte is about the advantages and annoyances of filtered indexes.  We’ll talk about
<ul>
 	<li>What they are</li>
 	<li>Where (and why) they’re useful</li>
 	<li>How to create one</li>
 	<li>How to convince the optimizer to use them</li>
</ul>
<h1>What is a filtered index?</h1>
<i>“A filtered index is an optimized nonclustered index especially suited to cover queries that select from a well-defined subset of data. It uses a filter predicate to index a portion of rows in the table. A well-designed filtered index can improve query performance as well as reduce index maintenance and storage costs compared with full-table indexes.”</i>

It achieves these wonderful feats mostly by being smaller than a non-filtered nonclustered index which must include one row for every row in its parent table – even when the indexed column(s) is NULL.  Remember – <b>smaller is faster!</b><b></b>

The other big benefit of filtered indexes is filtered stats which might get you better query plans.  Since the stats for a filtered index are built against only the data in the index they can be more accurate (granular) than stats built against an entire table’s data.

&nbsp;
<h1>When might you want to use one?</h1>
<ul>
 	<li>When indexing a column with a lot of NULL values.  Here’s an example from AdventureWorks:</li>
</ul>

[sql]CREATE INDEX ix_MiddleName ON Person.Person ( MiddleName );

CREATE INDEX ix_MiddleName_NotNULL ON Person.Person ( MiddleName )

WHERE MiddleName IS NOT NULL;[/sql]

<a href="https://msdnshared.blob.core.windows.net/media/2018/03/image328.png"><img width="486" height="98" title="image" alt="image" src="https://msdnshared.blob.core.windows.net/media/2018/03/image_thumb306.png" border="0" /></a>

&nbsp;

The filtered index is 32% smaller than the one with all the NULLs.
<ul>
 	<li>When you want to facilitate access to subsets of data:</li>
</ul>
&nbsp;

[sql]CREATE INDEX ix_OnlineOrderFlag ON Sales.SalesOrderHeader ( OnlineOrderFlag );

CREATE INDEX ix_OnlineOrderFlag_0 ON Sales.SalesOrderHeader ( OnlineOrderFlag )

WHERE OnlineOrderFlag = 0;[/sql]

<a href="https://msdnshared.blob.core.windows.net/media/2018/03/image329.png"><img width="503" height="98" title="image" alt="image" src="https://msdnshared.blob.core.windows.net/media/2018/03/image_thumb307.png" border="0" /></a>

&nbsp;

Again, the filtered index is much smaller.
<ul>
 	<li>Of course, useful indexes are seldom defined against a single column.  Perhaps you need to “cover” different sets of columns depending on which Status you’re querying for:</li>
</ul>

[sql]CREATE INDEX ix_Status_A ON dbo.Products ( Col1, Co2 ) INCLUDE ( Col4, Col7, Col8 ) WHERE Status = 'A';

CREATE INDEX ix_Status_F ON dbo.Products ( Col1, Co2 ) INCLUDE ( Col7, Col9, Col13 ) WHERE Status = 'F';

CREATE INDEX ix_Status_X ON dbo.Products ( Col2 ) ( Col3, Col9, Col12 ) WHERE Status = 'X';[/sql]

&nbsp;
<ul>
 	<li>You might also want to filter on various date ranges, but know that this is a bit trickier to implement since you <u>can’t</u> do something like the highlighted code:</li>
</ul>

[sql]CREATE INDEX ix_PONumber_DueDate ON Sales.SalesOrderHeader (PurchaseOrderNumber)

INCLUDE ( DueDate, TotalDue )

WHERE Status = 4 AND DueDate &gt; DATEADD(DAY, -5, SYSDATETIME());[/sql]

&nbsp;

Instead, you’d have to create the index using a literal (<b>deterministic</b>) date:

&nbsp;

[sql]CREATE INDEX ix_PONumber_DueDate ON Sales.SalesOrderHeader (PurchaseOrderNumber)

INCLUDE ( DueDate, TotalDue )

WHERE Status = 4 AND DueDate &gt; '20180101';[/sql]

&nbsp;

Creating an index with a somewhat larger date range can still result in an index that’s significantly smaller its parent table.  You can get more specific when you run the queries that will use the index.  Having a somewhat broader range allows you to update the index less frequently as your target date range moves.  You might also create multiple different filtered indexes having differing date ranges on a single table.
<h1>So, how do you create one?</h1>
<a href="https://docs.microsoft.com/en-us/sql/t-sql/statements/create-index-transact-sql">CREATE INDEX (T-SQL)</a>

<a href="https://docs.microsoft.com/en-us/sql/relational-databases/indexes/create-filtered-indexes">Create Filtered Indexes</a>

&nbsp;

It’s pretty straightforward, you just tack a <b>WHERE</b> clause onto your <b>CREATE INDEX</b> statement as in the examples above.  The expressions can only use the simple comparison operators ( =, &gt;, &lt;&gt;, <i>etc</i>.)  and a handful of other constructs (IN, IS NULL, IS NOT NULL).  Here are a few more examples:

[sql]WHERE StartDate &gt; '20000101' AND EndDate &lt;= '20000630'

WHERE ComponentID IN (533, 324, 753)

WHERE StartDate IN ('20000404', '20000905') AND EndDate IS NOT NULL[/sql]

&nbsp;

Note that the data from the columns in the WHERE clause isn’t stored in the index.  If you need those values, use the column in the key or included column list.

Here are a few additional limitations…
<ul>
 	<li>You can’t create a filtered index against a view (although a view can leverage filtered index defined on a table)</li>
 	<li>Errors will occur if column data is implicitly or explicitly converted</li>
 	<li>You cannot reference a computed column, a UDT column, a spatial or hierarchyID data type column</li>
 	<li>Filtered indexes do not apply to XML indexes and full-text indexes</li>
</ul>
&nbsp;
<h1>Why won’t the Optimizer use my filtered index?</h1>
In a word – <b>parameterization</b> – and it actually makes a lot of sense when you think about it in terms of query plan reuse (see this great <a href="https://blogs.msdn.microsoft.com/bartd/2018/02/08/filtered-indexes-and-forced-parameterization/">blog post</a> from Bart Duncan).

Queries are parameterized to allow reuse of cached plans.  This is generally good for performance – barring the occasional case of parameter sniffing.  Consider the following queries:

[sql]SELECT * FROM Person.Person WHERE LastName = N'Anderson';

SELECT * FROM Person.Person WHERE LastName = N'Baker';

SELECT * FROM Person.Person WHERE LastName = N'anderson';[/sql]

&nbsp;

To the Optimizer they’re all different queries.  We’ve got 2 different last names and a difference in capitalization.  Plans for all 3 queries will be separately compiled and cached, and we don’t get to leverage plan reuse.

If, instead, that literal last name value was cached as a <b>parameter</b>, it would remove the variability and allow plan reuse - for any value of the parameter.  We can achieve this using a stored proc:

[sql]CREATE PROC getPersonData ( @LastName VARCHAR(50) )

AS

SELECT * FROM Person.Person WHERE LastName = @LastName;

GO[/sql]

&nbsp;

<u>Or</u> let SQL help us out with <b>auto-parameterization</b> - SIMPLE or FORCED.  <b>Simple</b> auto-parameterizes simple T-SQL (like the queries above) and <b>Forced</b> parameterizes all queries.  Which is in use when set using ALTER DATABASE.  Simple is the default setting.  When auto-parameterization is in use you’ll see something like this in an <u>Actual</u> Query Plan:

[sql]SELECT * FROM Person.Person WHERE LastName = &lt;b&gt;@1&lt;/b&gt;;[/sql]

So, now we’ve got a plan cached for reuse – good for any parameter value.  All’s well, right??  Wrong.  This is where we butt heads with the Optimizer over use of our filtered index.

If a plan were to be cached using a filtered index for, say purchase orders from 2017, that plan is useless if someone later submits a query for data from 2016 or 2018 – date ranges not covered by our filtered index.  So, the optimizer simply doesn’t use it.  The Optimizer needs to cache a plan that’s good for a wide range of parameters – not just 2017 values.

&nbsp;
<h1>Coercing the Optimizer to do our bidding</h1>
Here are a few workarounds to you can try to encourage the Optimizer to use your filtered indexes:
<ul>
 	<li>Add literal values that align with your filtered index.  These will have to updated periodically if you modify your filtered indexes – say to follow a date range.</li>
 	<li>Query a view that’s based on literal values that align with your filtered index then query the view instead of the base table.   You can update the view periodically to cover a new date range that also aligns with a filtered index.</li>
 	<li>Use a RECOMPILE hint – this prevents the plan from being cached, reuse isn’t an issue, so the Optimizer is OK with using the filtered index.</li>
</ul>
&nbsp;

In the attached script we’ll create a filtered index then look where it’s used (or not) and explore the workarounds listed above.

Happy Exploring!]]></content:encoded>
    </item>
  </channel>
</rss>