Ploybase allows for a high performance integrated query across Hadoop and relational data using two key components - External Tables and the HDFS Bridge.

Here is some example SQL that is accessing ClickStream data stored in Hadoop and additional URL data stored in a dimensional table in the PDW.

SELECT
url.description

FROM hdfsClickStream cs

JOIN UrlDescription url

ON cs.url = url.name

WHEREcs.url = 'www.cars.com'

How does it work?

1. Polybase uses an internal representation of data structure stored in PDW metadata called an External Table to see tables in Hadoop. Once a Hadoop data source has been defined as an External Table the user can access the table in the same fashion as a relational table.
2. To satisfy a query Polybase will import/export Hadoop data to and from the PDW to take the best advantage of the available computing power.  This data transfer will be performed by the individual PDW Compute Nodes communicating directly with the Hadoop HDFS nodes for massively parallel data transfers.
3. Results are brought back (also in parallel) to the PDW and are temporarily (or permanently) stored on PDW. The PDW compute nodes complete processing and results are returned.  

External Tables

An EXTERNAL TABLE structure is what imposes structure upon the Hadoop data and identifies its location to PDW. The CREATE EXTERNAL TABLE statement defines the structure of the data being accessed and in the WITH clause locates the data in HDFS and specifies any formatting options. This metadata is stored on the PDF.  External Table metadata can be queried using the DMV sys.pdw_external_tables. External tables also appear in SSMS object browser.

CREATE EXTERNAL
TABLE hdfsClickStream (

    url varchar(50),

    event_date date,

    user_IP varchar(50)

)

WITH (LOCATION = 'hdfs://192.168.2.154:5000/tpch1GB/click.txt',FORMAT_OPTIONS ( FIELD_TERMINATOR = '|')

)

Once your external table is defined you may query it directly with TSQL and perform joins to PDW tables

SELECT url.description

FROM hdfsClickStream cs

JOIN UrlDescription url

ON cs.url = url.name

WHERE cs.url = 'www.cars.com'

HDFS Bridge

The HDFS Bridge provides an abstraction layer over HDFS. HDFS Bridge has been added to the Data Movement Service (DMS) that runs on each PDW node. This is what provides for the massively parallel data transfers.  This allows the HDFS data nodes to be queried concurrently by each compute node. Data moves in parallel directly between PDW Compute nodes and HDFS nodes. 

PolyBase was created in Jim Gray Systems Labs by  David DeWitt, former Professor Emeritus of Computer Sciences.  Dr. DeWitt is known for pioneering research in parallel databases, benchmarking, object-oriented and XML databases.  

I highly recommend checking out the lab's web site at http://gsl.azurewebsites.net/Projects/Polybase.aspx

And if you want to watch a good session tune in to David DeWitt's PASS 2012 Day 3 Spotlight Session  (Day 3 Spotlight Session with David DeWitt).

Book Review - Hadoop: The Definitive Guide, Third Edition (May 2012)
By Tom White

Copyright 2012
Published by O’Reilly Media Inc
ISBN: 978-1-449-31152-0

Overview
Organized into sixteen chapters and three appendices, Hadoop: The Definitive Guide, provides a comprehensive and detailed guide to the Hadoop ecosystem.  The first three chapters provide an overview and history of the Hadoop project and introduce the two primary components; HDFS and Map Reduce. The following chapters focus in great depth on the architecture of HDFS and Map Reduce. These chapters build upon the introductory chapters and dive deeper into topics such as architecture, availability, compression, file systems, building Map Reduce applications, jobs and tasks. The core components of Hadoop are HDFS and Map Reduce and and they are covered by the author in a progressive and digestible format.

Each chapter provides in addition to description, technical details and recommendations, working examples that build chapter upon chapter to walk you through simple illustrative examples of each of the concepts. The examples are well presented and easy to understand and consistently use the data and use cases from previous chapters so that the reader does not need to comprehend a new use case for each example distracting focus from the example’s message.

There are two chapters devoted to configuring and operating a Hadoop cluster and these are augmented by the three appendices that cover installation and prepping for the example code.  Working through the example set up in combination with these chapters should prepare the reader for their own Hadoop implementation.

Pig, HBase and ZooKeeper are addressed in later chapters but are not covered in the same depth as HDFS and Map Reduce. Each of these additional tools deserves its own reference and there are many available.  If you have covered the previous chapters the author’s introduction and examples to these three tools will get you started. 

 
The table of contents and index are well executed and topics are easy to find. The example code is available online which is something that I always appreciate.  Throughout the book the author provides links to the latest documentation online, hints and, best of all, Gotchas to look out for.

How I Read this Book
This is a lot of information to take in and if you are new to Hadoop as I was I would recommend viewing an overview video or two online to become familiar with all the vocabulary before diving into this book.  I got a LOT out of this book by tackling it in two passes; First, I read it all the way through but just glanced over the examples – understanding the use cases, but not getting all wrapped around the Java. It has easily been 10 years since I last developed anything in Java, so I postponed my personal syntax journey.  In my second pass through the book I focused on working and understanding the examples.  By reading first and then working the examples I found that I had more focus on understanding the details and concepts by reading with an uninterrupted flow. Following with a “working the examples” pass allowed me to review and reinforce concepts with example activities. I tackled this material using a “lecture then lab” aproach that works well for me.

Hadoop: The Definitive Guide is an excellent learning tool and reference for Hadoop.  I should be required reading for anyone interested in working with Hadoop.

Here are some quick links to the data storage options available on Windows Azure:

SQL Database (formerly known as SQL Azure)

This is Software as a Service (SaaS) serving up a full featured relational database engine.  SQL Database is based on SQL Server so if you are already working with SQL Server you will find SQL Database a familiar place.  It does not support all the features of SQL Server at this time, but more and more features are being added with each release.

Read more here.

 [Aside: If you need full blown SQL Server in Windows Azure please take a look at SQL Server on Windows Azure VM].

Azure Storage

Table

This is NoSQL storage. This is a key/value pair implementation. Don't be fooled by the name this is not relational storage. You will not be doing any complex queries against it, but it is excellent storage for high speed simple typed data. There can be a LOT of data here. This scales very well and a single table can hold 1TB of data.

Blob

Binary Large Object storage.  Unstructured binary data, good for video, images, big binary data.  A single blob can be as large as 1 TB.

Both Blob and Table storage are ISO 27001 certified managed services. Both can scale to massive amounts of data and both are accessible from REST and managed APIs.

More details here.

Hadoop on Azure (Isotope)

Although not categorized as storage per say, in Windows Azure it is found as part of Business Analytics. I want to add it to this summary anyway since one of its components (HDFS) is data storage.

The format definition; Apache Hadoop core provides data storage with the Hadoop Distributed File System (HDFS) and a MapReduce programming model to process and analyze in parallel the data stored in this distributed system.  Apache™ Hadoop™-based Services for Windows Azure is a service that provisions and deploys Hadoop clusters in Windows Azure.

Hadoop on Azure is currently in Customer Technical Preview (CTP). I'm lovin' it. I'll be bloggin more about it soon.

In the meantime, read about it here.

Hyperthreading and SQL Server

I am asked about hyperthreading all the time. People have a lot of different experiences with hyperthreading and SQL Server.  Some folks think it is wonderful and others never, ever enable it on a SQL Server.  I have seen it help  performance and I have seen it hurt performance. 

Hyperthreading is the SQL Server poster child for "it depends".   

Hyperthreading is a technology that provides some additional processing power by dividing a single core into two logical processors.  The thinking is to push even more work through the physical CPU by having more threads ready and reducing the time the CPU spends waiting for a ready thread. In a workload where the physical processors are already saturated then implementing hyperthreading can actually reduce the workload. Workloads with high levels of contention can also experience poor performance using hyperthreading. 

How hyperthreading will behave in your environment is dependent upon;

• Your workload
• Windows Server OS version
• SQL Server Version
• Physical CPU chip/chipset
       including cache size

Most environments running multiple SQL Servers will be running on any number of physically different servers with different CPU configurations.  Your SQL Server workload should be tested with and without hyperthreading on the specific server upon which it will be running in production.  This is a case where unless the servers in the test environment are exactly like the servers in production, down to the CPU chipset and configuration, the testing will not be valid. Repeatable test plans should be constructed and executed to gather metrics to quantify the use or disabling of hyperthreading for your environment.

Here are a few broad guidelines on hyperthreading;

• Turn off hyperthreading if the logical CPU count that would result exceeds the number of CPUs supported by your OS or other software.  Example: Windows Server 2008 R2 supports up to 64 logical  processors, If you have a 40 core server (4, 10 core processors) hyperthreading would result in 80 logical cores. This is 16 cores greater than the OS will support. At start up those 16 logical cores will not be initialized and will be idle.  That is 8 physical cores of processing power not in use.

• Turn off hyperthreading for SQL Server 2005 (in most cases). Some of the synchronization primitives do not see hyperthreading for what it is and interpret it as a multi core processor and some soft NUMA memory behaviours may occur that impact performance. This may be seen when processes execute that scan through large sections of memory. They may not affect your workload though so quantify your choice with testing.

• Test hyperthreading with your workload for SQL Server 2008, 2008 R2 and 2012.  Hyperthreading is fully supported within the SQLOS without issue, but the nature of your specific workload may create issues related to I/O or read in-efficiency.  The results may be seen as excessive CXPACKET waits that are due to the timing of true CPU threads vs.hyperthreading CPU threads.   High CXPACKET waits may be observed even when MAXDOP is set to 1. 

Power Options and Intel Turbo Boost Technology

When Intel first introduced performance scaling in their chips controlled by a power option in Windows many people were dismayed to find their SQL Server's CPU performance throttling down when Windows deemed the system "not busy".  The default Power Option for Windows is "Balanced" which allows the processor to save energy by lowering the core frequency below the base frequency saving energy and lowering CPU performance.  I came across some additional detail on this behavior and have provided it below. 

The following is an excerpt from Performance Tuning Guidelines for Windows Server 2008 R2 (http://msdn.microsoft.com/en-us/windows/hardware/gg463392.aspx) 

Today’s modern processors enable scaling of performance and power based on the current activity on the system. The different performance states are dynamically managed by Windows in conjunction with hardware and platform firmware to respond to varying workload requirements. . The 3 default power plans exposed by Windows provide varying tradeoffs of performance vs. power consumption. For example, if the High Performance power plan is selected, Windows places the system in the highest performance state and disables the dynamic scaling of performance in response to varying workload levels. 

Servers hosting SQL Server should always be configured for High Performance. (my comment) 

To change a power plan:

1. Click on Start and then Control Panel
2. From the list of displayed item under Control Panel click on Power Options, which takes you to Select a power plan page (If you do not see Power Options, type the word 'power' in the Search Control Panel box and then select Choose a power plan.) .
3. By default, the option to change power plans is disabled. To enable this, click the Change settings that are currently unavailable link.
4. Choose the High Performance option
5. Close the Power Option window. 

Intel Turbo Boost Technology

Under some configurations and workloads, Intel® Turbo Boost technology enables higher performance through the availability of increased core frequency (overclocking). Intel® Turbo Boost technology automatically allows processor cores to run faster than the base operating frequency if the processor is operating below rated power, temperature, and current specification limits. 

Turbo Boost should be enabled for servers hosting SQL Server. (my comment)

Windows Server 2008 R2 Processor Performance Boost Policy

Intel Turbo Boost Technology is a feature that allows Intel processors to achieve additional performance when it is most useful (that is, at high system loads). However, this feature increases CPU core energy consumption, so Turbo Boost is configured based on the power policy that is in use and the specific processor implementation. Turbo Boost is enabled for High Performance power plans on all Intel processors and it is disabled for Power Saver power plans on all Intel processors.  TurboBoost is disabled on Balanced power plans for some Intel processors. For future processors, this default setting might change depending on the energy efficiency of such features. To enable or disable the Turbo Boost feature, you must configure the Processor Performance Boost Policy parameter. 

The Processor Performance Boost Policy is a percentage value from 0 to 100. The default value of this parameter is 49 percent on Balanced plans and 0 percent on Power Saver plans. Any value lower than 50 disables Turbo mode on some current Intel processors. To enable Turbo Mode, set this value to 50 or higher. 

The following commands set Processor Performance Boost Policy to 100 on the current power plan. Specify the policy by using a GUID string, as shown below:

Powercfg -setacvalueindex scheme_current sub_processor 45bcc044-d885-43e2-8605-ee0ec6e96b59 100

Powercfg -setactive scheme_current 

Note that you must run the powercfg -setactive command to enable the new settings. You do not need to reboot the server.

For more detail please review the relevant portions of Performance Tuning Guidelines for Windows Server 2008 R2 (http://msdn.microsoft.com/en-us/windows/hardware/gg463392.aspx). 

Resources:

Intel Whitepaper: Intel Turbo Boost Technology in Intel Core Nehalem Processors

http://download.intel.com/design/processor/applnots/320354.pdf?iid=tech_tb+paper 

Kevin Kline with some thoughts on hyperthreading on SQL Server 2005

The Perils of Hyperthreading on SQL Server

http://www.sqlmag.com/article/sql-server/the-perils-of-hyperthreading-on-sql-server 

 Slava Oks on hyperthreading on SQL Server 2000

Be aware: To Hyper or not to Hyper

http://blogs.msdn.com/b/slavao/archive/2005/11/12/492119.aspx

Latte Art

If you are planning on building a Private Cloud and virtualizing your SQL Servers there are some new skills you MUST acquire.  In addition to all the skills you need to effectively manage your SQL Server instances and databases now you will need to be able to effectively manage the virtual environment that is hosting them.  Even if you work in an environment where the management of the virtual environment is not under your control you need to know how it works.

You will thank me for making you learn this.

Some of the new tasks for managing a virtualized SQL Server Environment

 Windows Hyper-V and Live Migration

Learn how to build and configure a Hyper-V environment with SQL Server guests. Learn this by doing it yourself. Learn how to create a guest VM, configure the CPUs, Memory, Networking and Storage. Learn about Cluster Shared Volumes (CSV).  Know what Pass-Through vs. Fixed disks are and why Dynamic disks are not a good choice for SQL Server storage. Next, install SQL Server on it and install some test databases. Create a SQL Cluster. Implement AlwaysOn Availability Groups.  You will need to know how the hypervisor manages memory and CPU and what the various configuration options do.  You will need to know how each configuration option works and how it effects SQL Server. Practice cluster failovers and Availability Group failovers and Live Migration moves. Learn how to make them work in harmony. Push all the buttons, turn all the knobs.

"Planning, Implementing and Supporting SQL Server Virtualization with Windows Server 2008 R2 Hyper-V and Live Migration" http://technet.microsoft.com/en-us/library/ff898403(v=sql.105).aspx

“Running SQL Server 2008 in a Hyper-V Environment – Best Practices and Performance Recommendations” http://sqlcat.com/sqlcat/b/whitepapers/archive/2008/10/03/running-sql-server-2008-in-a-hyper-v-environment-best-practices-and-performance-recommendations.aspx

Managing the Environment

System Center 2012.  Yes, you can try and monitor everything without using any tools other than your own scripts and PowerShell but why would you?

"System Center 2012 Capabilities Overview" http://www.microsoft.com/en-us/server-cloud/system-center/datacenter-management-capabilities.aspx

Monitoring your virtual environment

System Center Virtual Machine Manager and Operations Manager with the right management packs will make your life much easier. Like learning to monitor SQL Server performance you need to understand what to monitor on both the host and the guest and what the values mean. 

"Virtualization" http://technet.microsoft.com/en-us/sqlserver/Video/gg429826

"System Center Virtual Machine Manager 2012: VMM Gets Major Upgrade" http://technet.microsoft.com/en-us/magazine/hh300651.aspx

System Center Monitoring Pack for System Center 2012  - Virtual Machine Manager http://www.microsoft.com/en-us/download/details.aspx?id=29679

VM Templates

System Center Virtual Machine Manager will allow you to create guest machine templates so that you do not need to manually craft each and every precious SQL Server by hand.  You will need to master not only the VM template abilities of SCVMM but now is the time to dig into sysprep for SQL Server if you have been putting this off.

"About Virtual Machine Templates" http://technet.microsoft.com/en-us/library/bb740838

"Install SQL Server using SYSPREP" http://msdn.microsoft.com/en-us/library/ee210664.aspx

"SQL Server 2008 R2 Sysprep Step by Step" http://blogs.msdn.com/b/petersad/archive/2009/12/17/sql-server-2008-r2-sysprep-step-by-step.aspx

SCVMM Self Service Portal

The wisdom of allowing just anybody to create a SQL Server whenever they want may be questionable, but you need to get to know the SCVMM Self Service portal. Using this feature you can quickly provision new servers. The features supporting the self service portal also provide for usage tracking and charge back.  Choosing to implement charge back is another heated debate but either way you go the Self Service portal will allow you track SQL Server usage and have some insight into where all that capacity is going.

"Virtual Machine Self Service" http://technet.microsoft.com/en-us/library/bb740810

Backup and Restore - Data Protection Manager

Work with DPM and practice backing up and RESTORING your virtual environment.  See how the DPM recovery interacts with your SQL Server restore and recovery plans and update your recovery plans accordingly.  Learn how to recover from a wide variety of failures and practice, practice, practice.  It is a lot easier to learn how to recover successfully and document the recovery before the next disaster (I say this a lot).

"DPM: Managing Hyper-V computers" http://technet.microsoft.com/en-us/library/hh757970

It is important for you, the SQL Server professional, to know how to host SQL Server on a Hyper-V environment.  The virtualization specialists at your company may not be as well versed in the needs of SQL Server as you are and it will be your responsibility to make sure that the virtualization of your SQL Servers is performed to provide the best performance, availability and recovery for your clients. Virtualizing SQL Server is coming of age and there are more skills to learn to provide your customers with a reliable "it just works" SQL Server environment.

This T-SQL Tuesday is being hosted by Nigel Sammy. Enjoy.

Ever build a Slipstream install of SQL Server?  All that manual cut and move and pasting of little binary files here and there and everywhere it seems. Always felt so hands-on. Always wondered if it would actually work. Always had to update the thing.

Don't get me wrong it was a HUGE time saver if you were building several servers in the shop, but it did take a bit of extra time and testing. This was also a bit of a hidden feature. Not intentionally hidden, but not as well known as you would think. I know of a LOT of DBAs that are blissfully unaware of this installation option/technique. Bit of a shame.

Product Update

SQL Server 2012 improves and extends the Slipstream concept by adding Product Update functionality right into the SQL Server installation process.

Product Update is on by default and has the ability to search Microsoft Update, use Windows Server Update Services, or a folder or network share for SQL Server updates.  Any updates found get integrated into the setup process. No more building Slipstream installations. No more running the install and then the SP and then the CU. No more forgetting entirely to install service packs and updates.

By default SQL Server 2012 install will use Windows Update on the client and using its configuration (either Microsoft Update or WSUS) find whatever is the latest update; Service Pack, cumulative update (CU), or whatever combination of these is the latest patch level for SQL Server.

But what if you don't want the latest CU? What if the latest is not approved for use yet in your organization? What if I need to install the RTM? Never fear there are always choices.  If you just run the setup without changing a thing it will use Windows Update to retrieve any updates either from WSUS or Microsoft Update and if any exist the install will stop and list all the updates it has found and allows you to opt out of the updates entirely. What about that second scenario? What if your organization very carefully approves updates and has a prescribed CU level for all SQL Servers? Then as each CU is approved for use it should be downloaded and placed in a common network share. Then SQL install can utilize the new UpdateSource parameter to search in the specified network share for SQL updates.

The Product Update feature by default is enabled and uses the client's Windows Update settings.  If you wish to either disable Product Update entirely or you wish to seek updates from a folder or network share then you will need to specify input parameters to the setup process.

There are two parameters;

• /UpdateEnabled=True|False - allows you to turn off the Product Update feature
• /UpdateSource="MU"|local folder path or UNC path containing downloaded updates

There are three ways to specify input parameters for SQL Server 2012 setup;

1. Command line installation
2. Place the parameters in a ConfigurationFile.ini and start the install from the command line specifying that configuration file
3. Modify the DefaultSetup.ini file in /x86 or /x64 folder at the root of the installation media

Which you choose will have a lot to do with your organization. If everyone uses the same installation media, modifying
DefaultSetup.ini may be the best choice.  If everything is scripted, placing the parameters right in the script or in a configuration file used in the script may be the best
choice. 

Not many updates for SQL Server 2012 yet <wink> but this new feature has the potential to streamline your installation process, reduce the complexity of your SQL Server
installations and reduce the risk of updates not being applied properly.

More can be found here.

This weeks photo is from Jersey City,NJ looking across the Hudson River

(Scene One: Office of the DBA)

Enter: Friendly SQL Adventurer ready to help you be more pro-active in your management of your enormous SQL Server landscape.

Has your company invested in a database monitoring tool?  Something that will give you "live" performance information and alert you when specified thresholds have been exceeded?  Something that is watching all of your SQL Agent jobs and will let you know if one is running too long or has failed?  Something that allows you to configure all kinds of different profiles and classes of servers and levels of service?

You have one of these - wonderful!

Have you configured it for your environment?

Yes? Good, let's take a look…

Umm..

Okay, I see you have adjusted the performance alert thresholds….

So, you really only want to be paged when the CPU usage gets over 98%?

Oh, you  got tired of getting an alert for every little spike.

Did you know that you can configure this to only alert you when the threshold is exceeded for a specified duration?

No?  Okay, well you can configure this to alert you when CPU usage gets over 80% for 5 min.

Yes, that would be better.

So, I see you have divided your servers up into Dev, Test and Production. That's great.

I see they are all using the same performance profile.

You didn’t know you could create different profiles? Okay here let me show you….

You say you got this great tool and set it all up and added all your SQL Servers, but it alerted and paged you all the time so you adjusted some of the settings.

Kind of looks like you adjusted them so that they would never go off unless the building was on fire.

Ah, yes.  I don't like to be paged unnecessarily either.

Let's sit down and do a little analysis of your environment ...

Three Part Harmony

Before you even start to look at monitoring tools you need to know what you want to know and when you want to know it.

The first step to monitoring your environment is creating a detailed list of what you need to know.  For which servers this information is important?  What is the priority when everything is going wrong at once? Other environmental factors will need to be identified such as the lead time to allocate more disk space.

For example, remaining physical disk space for the production backup storage location is very important or your backups will start to fail. You'll want to be notified when it is getting low with enough lead time that you can do something about it.  If it takes six months to get more disk space then you need to know six months before your backup jobs start to fail that you need more space.

Generally, performance problems are more important to production environments then to development environments. Adjust your thresholds accordingly.

You get the idea.

I believe that you should monitor all of your environments, even development.  The dev SQL Servers are important for the developers to be able to do their work - they are production to the developers. If they are not performing well you will need to handle it.  It is important to monitor test servers too.  You will be able to catch poorly performing components or badly configured SQL Servers before they get to production.  If your test environment is supposed to be just like production that should include the monitoring.  This will allow you to take the performance you saw in test and compare it to what is being observed in production.

Take the time to identify each SQL Server - server, instance and database - and detail how it should be monitored, what should be monitored, the urgency, and who will be responsible for responding to issues.

Now, choose a tool. And here is the important bit…

Learn how to use it.

All of the available tools do a lot of things, but not by magic. You need to make the investment in learning how to fully utilize the tools you have chosen.  I know we all want to find a monitoring tool that will just be magical and solve all of our monitoring and alerting problems with little or no effort or configuration.

In Tune

Any monitoring tool will come with a set of default monitoring parameters. Most tools you can just install, let them discover your SQL Servers and let 'er rip!  Your inbox will be full in no time. 

Go to that work you did before you purchased the tool and configure it to work for you and proactively alert to your specifications. 

Monitoring tools are VERY worth the investment. Realize that the investment is not just in dollars spent to acquire the tool but in analyzing your environment to identify HOW and WHAT to monitor and investing in developing the skills to make full use of the tools you have chosen.

By thoughtfully designing a monitoring plan, becoming skilled with your chosen tool set, and configuring for your environment you will have created a monitoring environment that has solved all of your alerting and monitoring problems - just like magic.

and while waiting around for my flight I thought I would try and tackle fixing that 16 VLF segments of 16 GB each situation I described in my previous blog post.

You might just think "well that should be just like fixing the too many VLFs issue, just shrink it down and grow it back up". 

There is a bit of a hitch to this. That #1 VLF segment in the primary transaction log file is sticky.

You cannot get rid of that first 16 GB VLF without dropping the log entirely and I do not recommend that. Ever. Not if you love your data.

Here are the factors that make this first VLF segment "sticky";

1. SHRINKFILE will only shrink in VLF sized increments.
2. SHRINKFILE will  not shrink your transaction log to fewer than 2 VLF segments.
3. You cannot REMOVE the primary transaction log file (that is the transaction log file that was created during the original CREATE DATABASE statement)

Msg 5020, Level 16, State 1, Line 1

The primary data or log file cannot be removed from a database.

So, what can you do

Well, using the technique below you can get to a log file with the all but the first VLF down to the size you desire. Please note that I did not create a transation log of 256 GB to test this technique. I am not that patient.

Start by freeing up as much as you can by either running a transaction log backup (FULL) or CHECKPOINT (SIMPLE).

Run

DBCC LOGINFO(VLFTEST)

To see what you have to work with.  The number of rows indicates the number of VLF segments your log has.  Take note of where the lowest VLF of Status 2 is located.

When the first VLF segment is the only VLF with a status of 2 (like pictured below), shrink the primary transaction log file down to two VLFs

 [Just a note here.  Shrink can only release VLMs up to the "lowest" one. If your lowest #2 is down the list a ways it you will have to wait around for enough transactions to go through for the log for it to wrap back around to the front. Depending upon the workload of your system you may need to be patient (Yes, you could do something like create a temp table and INSERT and DELETE a bunch of data into it to get your transaction log to roll around, but this may introduce unwanted risk into a production system and waiting may be a better choice).]

DBCC SHRINKFILE (VLFTEST_LOG, 1)

You should be rewarded with 2 VLFs left

Now, remember that it will not let you have fewer than 2 VLFs so to trick it we will temporarily add a second transaction log file

ALTER DATABASE VLFTEST

    ADD LOG FILE (NAME=VLFTEST_LOG2,

                                  FILENAME = 'C:\Program Files\Microsoft SQL Server\MSSQL10_50.MSSQLSERVER\MSSQL\DATA\VLFDATA_Log2.ldf',

                                  SIZE = 100 MB)

Shrink the primary transaction log file again and it will get down to 1 VLF segment in the first file

DBCC SHRINKFILE (VLFTEST_LOG, 1)

Using a series of ALTER DATABASE MODIFY FILE statements grow primary transaction log file out to the size and VLFcount you desire (like 480 512 MB VLF segments plus that first 16 GB segment for last week's example).

And finally, remove that second log file shrink that second transaction log file with EMPTYFILE option to move everything back into the primary file.

DBCC SHRINKFILE (VLFTEST_LOG2, EMPTYFILE)

Now remove the unwanted second transaction log file

ALTER DATABASE VLFTEST

                    REMOVE FILE VLFTEST_LOG2        

You should be left with your desired number of VLF segments in the primary transaction log file with only the first segment being a segment of unusual size.

If anyone knows of a method to "resize" that first VLF segment that does not involve dropping the database or log in some fashion please drop me a line.

 256 GB transaction log file with 16 VLFs.  Doing the quick math that would make each VLF 16GB.  16 x 16GB VLFs = 256GB transaction log. (yes, that is a large transaction log)

All this guidance I see says 50 VLFs would be too many.

Okay, not too many VLFs, so we should be fine… Matter of fact, if 50 is too many isn't 16 awesome?

Well, no. 

If you have too few VLFs, and more importantly VLFs that are too big, you will see poor performance from any of the processes that need to process these big VLFs.  These include your transaction log backups, transactional replication and database mirroring.

Transaction log backups will suffer when a large VLFs is finally filled and inactive, ready for backup and clearing.  In the above example 16GB of log will have to backup and a 16GB chunk of log will need to be cleared for reuse.  How fast is your backup environment?

Transactional replication processes the transaction log using the Log Reader.  Only committed transactions are replicated so it has to process through 1 or more 16GB VLFs (depending on how large the transaction was). These very large VLFs can cause additional latency in your replication. 

Database mirroring sends to its mirroring partner committed transactions extracted by the logging system itself  within the Storage Engine. The logging system is also going to feel the burden of an enormous VLF.  

Now consider a system that is replicating and mirroring - ouch!

So how do you find the  size/count balance?

First, evaluate how many VLFs a transaction log has (DBCC LOGINFO) and run a sanity check and see if it is a reasonable number for the file size.

Both 16 VLFs for a 256GB transaction log and 39 VLFs for a 11 MB transaction log fail the sanity check.

Next,  how big is each VLF? This is also in DBCC LOGINFO.  You will notice that the segments are probably not all exactly equally sized and the individual sizes may vary quite a bit.  This is the nature of how transaction log files are sized and grow and there are excellent details on this in  Kimberly Tripp's  [blog|twitter]  blog here. 

Is the size too big for an efficient transaction log backup (or log reader) process?

I have had excellent success with VLFs sized around 500 MB, but for large transaction logs this may create too many VLFs. 

Taking Exhibit A, let's examine our options;

• Keeping our VLF count to 50 will give us VLFs around 5GB is size
• Creating 512 MB VLFs will create 512 VLFs

• A 1 GB VLF will give you 256 VLFs
• A 2 GB VLF will give you 128 VLFs

In this specific example the transaction rate, whether replication or mirroring is in use, whether clustering is in use, the frequency of the transaction log backups and the throughput of the transaction log backup process needs to be analyzed to identify the best VLF size/count combination for this environment. 

When all is said and done, okay, so there are more than 50 VLFs,  but each VLF is sized so that there are no performance issues with the transaction log backups and the number of VLFs is reasonable for the total transaction log file size and no where near the "so many it is painfully funny" realm that results in recovery and other problems.

Corrected 4/9/13: Thanks to Paul Randal of SQLskills.com (Blog | Twitter) for reading my blog and pointing out an error on how database mirroring gets its transaction log entries.  As usual every time I interact with Paul I learn something.

One of my favorite short flights is the flight from Seattle, WA to Portland, OR.

If the weather cooporates you can get views like this;

Mt. Rainier

There are a number of things I like about this flight.  First, I like airplanes with props - just something about them. Second, this flight is so short that as soon as you clear the mountains you start coming back down.  I don't know what the altitude gets up to but it is not very high. That leads to third,  I love looking out the window and seeing the world from above.  I grew up around here so I can name all the mountains and rivers and lakes along the way.  I am fascinated by the shape of the land and seeing the effects of vulcanism, wind, water and time on the landscape.

This flight was particularly nice.  Late afternoon the sun was just setting and the morning's snowstorm had just blown itself out..

Mt. Saint Helens (that's Mt. Adams in the background)

What are we going to see here?

In this blog I will be covering SQL Server technical topics focusing primarily on the relational engine, database design and development and query performance.  It will be driven primarily by the work I do as a SQL Server specialist in the Microsoft Operations Consulting team.  Microsoft classifies me as "non-location specific" which means I travel. All... The... Time. I have the opportulity to assist and consult all over the country for an incredible variety of SQL Server customers. My clients come in all shapes and sizes and so do their SQL Server environments.  One week I may be coaching performance monitoring to a brand new accidental DBA and the next week trying to eek out the absolute fastest data load for a 2 TB table with a team that has already tried everything they can think of. My job is not boring.

There will also be blogs like this one that are not technical at all.

Let the adventure begin!