Let’s pretend for a moment that you cannot get faster IO and your problem is data warehouse style table and queries.  The key here; not an OLTP workload.  If this is the case Clustered Columnstore Index (CCI) in SQL 2014 may be worth investigating.

 SQL Server 2012 introduced the non-clustered columnstore index which was met with great fanfare to everyone who tested and the few who were able to use it.  A limitation of the non-clustered column store is it rendered the table read only.  In SQL 2014 the Clustered Columnstore is updatable. 

What you need to know,

• Clustered Columnstore is the only index on the table
• Data is stored in an columnar format versus a row format
• Columnstore is highly compressed, unlike non-clustered column store index, your entire table may be reduced in sized from 2x-20x. 
  
• Each column is compressed into a Column Segment that has up to 1 million rows (1 column of 1 million rows, for simplicity i use 1 million rows for the column segment, but is is actually 1,048,576 rows, (1024*1024))
  
• A group of 1 million rows makes up a row group (the row group is made up from the columns, if your table is 10 columns, your row group would be made up of 10 columns by 1 million rows)
• Until you have inserted 1 million rows your data is kept in a deltastore
• If you bulk insert 102,400 they will go directly to the columnstore bypassing the deltastore
• CCI Archival_Compression you will gain even more compression, this is very good for older data that is accessed less often
• Only columns that are in the SELECT, WHERE predicate or/and JOIN and GROUP BY are read from disk thus allowing column segment elimination
• I/O for an entire row group is eliminated if the row segments are out of range of the WHERE predicate or JOIN
• CCI can be partitioned for easier management of your table
• If your query uses every column in the table, your performance may be worse, consider the queries accessing the table before blindly converting to a Clustered Column Store Index
• Once a column segment is created it is never physically changed, deletes are maintained using a delete bitmap
• Updates are a delete and insert, delete bitmap is updated to mark the old row and the new row will be inserted into the delta store
• A high number of updates may reduce overall query performance against the CCI table
• Batch Mode processing moves data between query execution operators in vectors (up to 1000 rows at a time) vs row mode processing which moves 1 row at a time
• Columnstore queries are highly parallel operations, they are not limited by k-groups, Columnstore operations do obey MAXDOP settings
• Alter Index Rebuild is used to physically delete rows marked for deletion in the delete bitmap
• Alter Index Reorganize will only move closed row groups in the deltastore into the columnstore
• SQL 2014 CCI can participate in an AlwaysOn availability group however it is not readable on the secondary
• If you columnstore index creation fails with out of memory error 8657, 8658, 701, 802, increase the resource governor default resource pool max memory per the SQL Server Columnstore Index FAQ for the creation of the index, do not forget to set it back when complete

To give you a sense of the data compression you may experience, the following is test telemetry data from a Microsoft application that came through the lab, and keep in mind the column data was relatively redundant so there was great opportunity for compression.

Looking at the SSMS table size report below, the extension on the table name indicates the level of compression based on size from a standard row store table to a Clustered Column Store Archive on a table with 1,111,997,989 rows and uncompressed size of 182GB.

*_RS = Row Store

*_RS_PC = Row Store Page Compressed

*_CCI = Clustered Column Store Index

*_CCI_Archive = Clustered Column Store Index with Archival_Compression

The following sample used a query test to get an idea of the potential performance for 1 execution of the query with a cold cache, and each query run on the same server with the same hardware configuration.    

(For this test I used a 2 processor 16 core, 128 GB RAM with SQL allocated 112GB, 2 mirrored 10k RPM disks for data file)

Columnstore Indexes Described

Using Clustered Columnstore Index

SQL Server Columnstore Index FAQ

SQL Server Columnstore Performance Tuning

SQL Server 2014 and HP Sets Two World Records for Data Warehousing

From the lock compatibility matrix you can determine that a Sch-M lock is in conflict with everything. It is not difficult to determine that if a transaction comes along after the Sch-M has been requested you may end up with a pretty significant blocking chain.  If you have a table that has a significant OLTP workload,  reindexes and partition switch can be a troublesome operation to schedule.

SQL 2014 addresses this with Managed Lock Priority (MLP).   MLP creates a low priority queue for both partition switch and for online index operations, MLP will not work with any other features in SQL 2014.  You will notice new WAIT_AT_LOW_PRIORITY syntax for ALTER TABLE and for ALTER INDEX. 

Here is what the result is when you use the new features of the syntax;

Using AdventureWorksDW2012 the syntax below will rebuild the AK_DimSalesTerritory_SalesTerritoryAlternateKey index on DimSalesTerritory will waiting for up to 15 minutes, then if the Sch-M lock cannot be gained, the DDL statement will terminate itself.  If the Sch-M lock can be taken immediately the statement will execute without wait.

ALTER INDEX AK_DimSalesTerritory_SalesTerritoryAlternateKey
ON DimSalesTerritory REBUILD
WITH ( ONLINE = ON ( WAIT_AT_LOW_PRIORITY (MAX_DURATION = 15 minutes, ABORT_AFTER_WAIT = SELF )))

The following statement will perform an online rebuild of just partition 5 wait 0 minutes (execute immediately) and kill all blockers.  You will also notice that a single partition can now be rebuilt online as well.  While the AdventureWorksDW2012 does not partition the DimSalesTerritory table, this is the syntax that would be used.

ALTER INDEX AK_DimSalesTerritory_SalesTerritoryAlternateKey
ON DimSalesTerritory REBUILD PARTITION = 5
WITH ( ONLINE = ON ( WAIT_AT_LOW_PRIORITY (MAX_DURATION = 0 minutes, ABORT_AFTER_WAIT = BLOCKERS)))

Since the DDL is waiting in a low priority lock queue any new transactions that come in after the DDL was submitted will not be blocked behind the DDL. 

One little nugget to keep in mind is that of you use the MAX_DURATION, this will prevent the transaction log from truncating from the point the DDL was submitted until it is executed.  For example, you can set the MAX_DURATION up to 71,582 minutes (1,193 hours or 49 days) unless you have the transaction log drive space to support 49 days’ worth of transactions, you might want to keep the MAX_DURATION a bit lower.

For a deeper dive and to see the new waittypes introduced for this feature review Improved Application Availability During Online Operations in SQL Server 2014.

Typically a SQL Server Cumulative Update release is not a noteworthy event, however SQL Server 2012 SP1 CU9 and SQL Server 2014 CU1 include a fix that is noteworthy from a from a CAT Lab perspective.  In short, if you have a NUMA server with 32 or more physical cores you may want to investigate this CU.

If you are seeing waittypes of SOS_PHYS_PAGE_CACHE from the sys.dm_os_wait_stats DMV or spinlocks of CMEMTHREAD or SOS_SUSPEND_QUEUE in the sys.dm_os_spinlock_stats DMV, I would highly encourage you to investigate the new CU sooner rather than later. 

FIX: Performance problems occur in NUMA environments during foreign page processing in SQL Server 2012 or SQL Server 2014

Cumulative update package 9 for SQL Server 2012 Service Pack 1

Cumulative Update 1 for SQL Server 2014

At the time of this posting the Trust center has provided compliance framework for ISO/IEC 27001:2005, SOC 1 and SOC 2 SSAE 16/ISAE 3402, Federal Risk and Authorization Management Program (FedRAMP), Payment Card Industry (PCI), Data Security Standards (DSS) Level 1, United Kingdom G-Cloud Impact Level 2 Accreditation, Family Educational Rights and Privacy Act (FERPA), and HIPAA Business Associate Agreement (BAA).

All of these can be reviewed from the Microsoft Azure Trust Center Compliance Page and the Microsoft Azure Trust Center Resources page.  Be sure to read the FAQ as well, it covers many of the questions I hear related to who has access to my data and how are Azure Security Audits actually done.

Other links:

Privacy authorities across Europe approve Microsoft’s cloud commitments

Microsoft’s Law Enforcement Requests Report

Responding to government legal demands for customer data

CAT has released In-Memory OLTP – Common Workload Patterns and Migration Considerations.  This paper was written from the experiences gained during the Pre-RTM phase of SQL 2014. 

Additionally Kalen Delaney worked with the SQL Product Group to write SQL Server In-Memory OLTP Internals Overview.  Kalen’s paper will also help you get a jump start using In-Memory OLTP with a few samples and walkthroughs to help you understand each component of In-Memory OLTP.

In-Memory OLTP – Common Workload Patterns and Migration Considerations. 

SQL Server In-Memory OLTP Internals Overview

Books Online Introduction to Memory-Optimized Tables

Good News Everyone!  SQL Server 2014 Standard Edition has new features for 2014.

1.  Increase in supported RAM to 128GB.

2.  Addition of Buffer Pool Extension (BPE).  Imagine BPE as a warm cache for the SQL Server buffer pool.  As clean pages are aged out of the buffer pool due to memory pressure, having to re-read them from storage causes an IO.  Many servers still have spinning disk supporting them be it internal drives, direct attached storage, SAN or network attached storage, an IO can become an expensive operation.  BPE will allow pages to be aged out from the buffer pool to a BPE file on SSD or Flash, thus creating a warm pool.

I will say, it is possible to create a BPE file on non SSD or Flash storage, SQL does not check, so it is possible to degrade the performance of your system by creating your BPE file on slow storage, in short, don’t do that.  BPE is only clean pages, so if the server loses power or goes off line for any reason, the BPE file will be cleared but no data is lost since these were all clean pages that exists on disk.

A SQL Server Standard Edition BPE file can be up to 4X the amount of RAM allocated to the SQL Server using Max Server Memory.

3.  Addition of Delayed Durability.  Delayed durability is essentially a lazy commit.  There is a lot to consider when deciding to use this.  The books online cover all the scenarios, but in a paragraph here are the things you need to know;

Delayed Durability can be implemented at the Database level, or the transaction level.  Writes to the log are asynchronous, that means your transaction can complete before the log record has been hardened.  This means more concurrent log IOs and less contention on the database log file, so if you are waiting on WRITELOG delayed durability may help.   However, your transaction has completed before your data has been hardened to disk, this means that you could suffer from data loss for any uncommitted data during a catastrophic event such as power loss.  So, think very carefully before you implement this.  Please review the section “When can I lose data” before implementing this feature at the database level or the transaction level.

Check the books online before implementing this and review the section on Delayed Durability and other SQL Server features.  For Standard Edition log backup, log shipping, transactional replication, crash recovery all may have behavior changes once delayed durability is enabled.

4. The Cardinality Estimator in Standard Edition has been updated as well, see my prior blog on the new CE for more details.

All of this can be found in Optimizing Your Query Plans with the SQL Server 2014 Cardinality Estimator

A re-designed cardinality estimator in SQL 2014 has been implemented to improve the quality, performance, and predictability of query plans.  The goal was for the QO to behave better!  In lab testing though there are a couple things I want to point out; If you have over engineered your queries e.g.  you exploit known goofy behavior or you are a frequent user of QURETRACEON to change the behavior QO for a specific query, you need to retest everything, I cannot stress that enough. 

If you upgrade and find this out after the fact, all you have to do to end your temporary nightmare is change the database compatibility level back 110, this will invoke the “70” Cardinality estimator.  This is good to know if you need to implement the old behavior in a hurry, like on Monday morning when your users come in and start hammering your brand new SQL 2014 upgraded instance.  Having the database in compatibility 120 will invoke the shiny new estimator in 2014.  

If for some crazy reason you end up with only the execution plan from a query as part of a later investigation, you can verify the estimator that was used by checking the execution plan XML and searching for “CardinalityEstimationModelVersion=” your choices are ‘70’ and ‘120’. 

You can also use Trace Flag 2312 to implement the new behavior on a single query under the old cardinality estimator (70) using the QUERYTRACEON or use Trace Flag 9481 under the new cardinality estimator (120) to force the old behavior for a single query.  Not confusing at all, right? 

Additionally, improvements have been made to the ascending key issue that was addressed by trace flags 2389 and 2390, if you have been using them be sure to test your workload with the new CE.  

We have just released a new paper authored by Joe Sack on all of the ins and outs of the new CE, happy reading!   

Optimizing Your Query Plans with the SQL Server 2014 Cardinality Estimator

SELECT db_name(er.database_id),

er.session_id,

es.original_login_name,

es.client_interface_name,

er.start_time,

er.status,

er.wait_type,

er.wait_resource,

SUBSTRING(st.text, (er.statement_start_offset/2)+1,

((CASE er.statement_end_offset

WHEN -1 THEN DATALENGTH(st.text)

ELSE er.statement_end_offset

END - er.statement_start_offset)/2) + 1) AS statement_text,

er.*

FROM SYS.dm_exec_requests er

join sys.dm_exec_sessions es on (er.session_id = es.session_id)

CROSS APPLY sys.dm_exec_sql_text(er.sql_handle) AS st

where er.session_id in

(SELECT distinct(blocking_session_id) FROM SYS.dm_exec_requests WHERE blocking_session_id > 0)

and blocking_session_id = 0

But, hope is not lost, all of the content is in the process of being moved to the MSDN SQLCAT site which actually has been a mirror for most of the content for the last 8 years.  If you go back to the first post, you will even find Mark Souza’s’ original post introducing SQLCAT and the Advisory team.  The downside of the mirrored site is some of the content did not get moved.   We had a couple of categories of posts that did not make it, one was the whitepaper introductions and links.  While all of the Whitepapers exist on MSDN download sites, when you search for a known whitepaper your search engine will frequently take you to the SQLCAT site which is now offline. 

To help solve this I put together a list of consolidated index of Whitepapers from the last several years, I did drop a few as they were way too old, but almost all of them do hold amazing value to the community.

SQLCAT.com consolidated index of Whitepapers

http://blogs.msdn.com/b/sqlcat/archive/2013/09/05/sqlcat-com-consolidated-index-of-whitepapers.aspx

SQL Server Customer Advisory Team

 http://blogs.msdn.com/b/sqlcat/

You asked for it Microsoft did it.  Billing per minute is pretty self-explanatory, prior to June 3 all IaaS compute hours billed by the hour and for the most part rounded up, no more, now you are billed per minute! 

However the best thing related to billing was the Stopped (Deallocated) VMs no longer incur compute charges.  This is very important for Dev/Test environments.  Many customers were exporting their VMs to a blob store nightly and over weekends to stop the charge per hour for a test machine.  Now you can stop the compute clock by stopping (Deallocating) the VM. If you only have your VM turned on for a few hours a day you will only pay for a few hours a day.  Keep in mind, you will still be charged for other resources that may be part of your deployment, for instance if you have 16 disks attached all at 50% capacity, you will continue to get billed for that as you are still occupying space.  This only applies to compute hours, I now this will make a lot of developers and testers very happy!

Shep

Windows Azure: Announcing New Dev/Test Offering, BizTalk Services, SSL Support with Web Sites, AD Improvements, Per Minute Billing

http://weblogs.asp.net/scottgu/archive/2013/06/03/windows-azure-announcing-new-dev-test-offering-biztalk-services-ssl-support-with-web-sites-ad-improvements-per-minute-billing.aspx