The other thing I want to lay out here is technology.  I LOVE technology.  New gadgets, tools, ideas, etc..., but I do feel that each new technology has boundaries of where that technology is good.   Take the new Electric Car, Great idea, if you want to go 100 miles, but if you're traveling across the country, probably not the "Tool" you are going to use.

Virtualization has been the craze in the last decade or so.  I personally think this technology is AWESOME, but just like above, it does have limitations.  Those limitations, in my opinion make it not always the best choice for a SQL Server.  Now I have to say "Not Always", because just like most things, there is not one solid answer.  

Let's walk through the "Why".  Why do I want to virtualize?  I want to reduce the number of boxes in my server room. Reduce the Power I consume, reduce the licenses I buy.   All of these things basically can be wrapped up into consolidation.   If I owned or paid the bill for a server room, I too would want to run it with the least cost to get the job done.   Here is where the issue comes in play.  What if I have 100 database servers that I want to Consolidate?   Well if I don't want to get any groups other than my platform teams involved, I could take all of those servers and virtualize them down to 5 physicals.   This sounds good, right?  If you look at this from a purely infrastructure/OS point of view, then yes, we have saved some money.  But could we have done this better if we looked at the Big Picture?

Take those same 100 servers.  What if I built a 5 node SQL Cluster?   What if I could consolidate all of the databases on those servers down to 6 instances that live on that cluster?  On the surface, we have accomplished the same thing here.  We went from 100 servers to 5, but I think we have done more than that.   If we include the DBA team, we can say they first consolidation/virtualization reduced the servers they manage from 100 to 100.  Wait, you mean that the DBA team still has 100 servers to manage?  Yep, since all I did was virtualize the existing servers, the only savings was from the Hardware and power they consume.  Even the backup folks need to still backup 100 servers.  That's still 100 licenses for Antivirus, Backup Agents, Security Monitoring, and Performance monitoring just to name a few.    Ok, the second one did what?  It reduced the Hardware and Power, but also reduced the number of SQL Servers from 100 to 6.  That means that I only have 6 Instances of SQL to watch and maintain.  Six log files, Six Error Logs, Six servers to backup, etc...   We also reduced the Agents that we need to buy.  

I have not even started to talk performance yet.  The absolute largest bottleneck or pain point I run into is I/O.  If I live on virtualized hardware, I can manage this, but from my experience, we don't.   I sit on virtual drives that are shared with File Servers, other DB servers, App Servers, and the OS's of all the other virtualized servers.   The typical setup is to have a very large LUN carved from a SAN and then assigned to my Virtualized environment, and then we take that LUN and carve it up even more.   Can you not do this, Yes!  I just don't see it in the field.  On the other side, how do we do I/O?   Well if it were me designing the system, we would have one main LUN that would be the drive letter, then we would have several Mount Points under that drive letter.  This way I would be able to separate Data, Log, Tempdb, Indexes, etc...  I would also have a monitor that would help me choose when a particular LUN was "FULL".   Not in terms of physical size, but in terms of performance.   This way we can separate those databases that can consume massive amounts if I/O, but not CPU or Memory.  

One other small detail here is that if we consolidated the second way, we also get High Availability.  Since we are in a cluster, we can have reduced down time for OS and SQL Patches.

Earlier I said there is not one solution for everything, so where and when does virtualization of SQL make since?  I feel they are in specific situations.   Consider having an old SQL 7.0 server sitting on 15 year old hardware.  You can't upgrade because the application hasn't been upgraded, and it's required for your company.   Take that thing and virtualized it!   This is a disaster waiting to happen.   If it's been running, Software is probably not the issue, but that hardware is going to start having issues, and when it does, getting parts is going to be even worse.   This is just one example, but there are others like this.

So here is my take on consolidation/virtualization.   Virtualization is a fantastic new technology, but it is not the Answer to EVERYTHING.   There are other tools, and technologies that could, and do make more sense to put in certain places.   If you are looking at a consolidation project, I would urge you to look at this project from different perspectives.  Think that you are the CIO and you need to consolidate and reduce costs.   I think my solution does both, and frees up some time for the DBA's to do things other than watch backups and error logs on 100 servers.  

Eric, @SQLPilot
Tweet

Since this is such a broad topic, I want to just start with the basics.   I'll review some of the settings in SQL and Windows, and then talk about what some of the pressures I see are, and how we can help reduce them.

First let's cover the Settings...

SP_Configure holds just a few of these settings.   Max Server Memory and Min Server Memory.  

Max Server Memory:
Max is kind of what you would expect.   It is the Maximum number that the Buffer Pool will grow to.   Please note I did not say it's the total Memory that SQL will consume, but just the Buffer Pool.   This is important as SQL can take more than this for other reasons.  Not EVERYTHING is in the Buffer Pool.   Things like Query Plan Cache, Memory for the Optimizer, Connections, CLR Objects, etc... all use memory that canbe outside of the pool.   Max Memory also does not limit other areas of SQL.  SSIS, SSRS, SSAS all are their own separate processes, and they are not controlled by this.   I know this may not sound ground breaking, but I still go on-site to customers who think this setting will control everything that SQL does.

Min Server Memory:
This is a setting I feel is very much not understood.   First and foremost, this DOES NOT pre-allocate memory.   SQL will NOT start up and grab this much ram.   This setting instead is a threshold.   Once SQL does grow beyond this number, it will not give it back to the OS.   Please re-read that again.   It does not give it up!    This is 0 by default, and although I have had several conversations on it, I personally don't see why this should be changed.    If it did pre-allocate it, then I could see a change, but if my server is under a Memory Attack, I would want SQL to release memory to save the OS.

AWE:
AWE is left over from the 32bit days, and why it is not required in 64bit, it does not hurt anything, so I always turn it on.

Lock Pages in Memory:
This is not a SQL setting, but it is one that has some old skeletons.  I will say that in SQL 2000, once this was set, SQL did not like to trim this memory, but that was then, this is now.   We DO want this to be set.   Why?   Here is the quick overview.   If I fill my SQL Servers memory with data, why would I want to page that data out to a page file?   I would not.   Most of us are going to have much faster disks for data files than for our OS Page file.   So if I wanted to remove the data from RAM, I would just rather throw it away and then reload it from the faster disks when needed again, not from the slower Page file.   If you are running into a Memory issue, you may have Max Server Memory set too high, and that needs to be changed, not this locking.

Okay, now we have just a few settings covered, let's talk about how to see this pressure, and what direction it might be coming from.   The tool I use here is Perfmon.   It is absolutely my #1 tool.  Not because it's a MS tool, but because it's easy to use, light (little if any impact to the server), on every windows system by default, just to name a few.   In Perfmon I want to look at memory on the box, as well as memory in SQL.  

• Memory: Available MB
• SQL Buffer: Free Pages
• SQL Buffer: Page Life
• SQL Buffer: Lazy Writes

These four are my starting point.   First, if there is less than a few hundred Megabytes free on the server?  If so, something is wrong.   Either I have too many processes running or Max Server Memory is set too high.   If OS Memory is ok, now we need to see if there is pressure inside of SQL.   The other three counters will help with this.   I'll dig into what each one does first.

Free Pages:
This is the number of free pages in the Buffer.  Remember there WILL always be free pages, but how many and how does that number change?  The number we are looking for is 640 free.  Below that, the engine will either try to grow the buffer in memory, or start tossing the old data out.   This is done through the lazywritter, which happens to be the other Perfmon counter we look at.

Page Life Expectancy:
This counter tells us how long a page has lived in memory when it's no longer referenced.   The ideal number is over 300 seconds, obviously, the longer the better.

Lazy Writes:
This is the process that is used to throw away old data in the buffer, if that buffer can't be grown to a larger memory footprint.  Our target here is less than 20/sec.  The lower, the better.

The combination of these three counters will help us see pressure inside of SQL Server.   Most DBA's will monitor Buffer Cache Hit Ratio.  This counter does not really help us.  If the counter is low, then there is a good chance there is an issue, but if it's high, it tells us nothing.  Yep, nothing.  This is why I don't look at it.   On the other hand, the three I laid out will, and they all work hand and hand.

If I have low free pages, and the buffer can't be grown, Lazy Writes will start removing data. Once the oldest data is removed, this will reduce our Page Life.  See the combination of the three help prove this.    Let's look at it another way.  What if Free Pages is low, but Page Life continues to grow and Lazy Writes is at 0?  Well, this just tells me that we are running queries, and growing the Buffer Pool.  In that case we could look at the Available Megabytes, and see that it is decreasing by exactly the same trend as SQL is taking memory.   Again the same counters, but this proved that we did NOT have internal memory pressure.

One other little tip I like to add here.  You need to rule out Memory pressure first.   If you have Memory issues, I would bet that you have or will have disk issues.  Once you have thrown out all of you nice old data in memory, we are at some point going to have to load it back in.   When that happens, we have to read from the very slow disks.  I know, I know, you are on a SAN, and you have the fastest disks in town, well that does not matter.  Disks are SLOW, even the fabulous Solid State drives (Like I have in my laptop) are slower than just re-using data that is already in memory.

The other thing I need to point out is WHY.  Why is SQL doing this?  SQL's not doing it; it's doing what you told it to do.  How so?  With your code!  T-SQL queries, Stored Procedures, Function Calls, etc...   Up to this point, our lives have been pretty easy.  We used Perfmon to track down where the problem is.  IF, and I stress here, IF the problem is in SQL, then we are going to have to look at code some place in order to treat the symptom.   Yep, I guess that means cracking the chest and looking at that evil stuff called code.

I'll get detailed about how to find this code and what to and what not to write in a future blog, but high level, you are going to need to look at your highest I/O queries, and start tuning.   Adding Indexes, maybe roll-up tables, etc...   Some absolute basics are...

1. Don't query data that you don't plan to use.  NO SELECT *!!
2. Always use a WHERE clause.
3. Create Non-Clustered indexes that SEEK the data, not scan.
4. Having a Clustered index is "Usually" a requirement.  (Not required, but should be for 99% of the tables out there.
5. Be careful with SELECT DISTINCT, its better not to duplicate the data in the first place.

I could go on forever there, but I have to stop.

To recap what we talked about here.  We focused on just Memory Pressure.  My primary tool for finding this is to use Performance Monitor (Perfmon).  First look at the OS level, and then dig into SQL.  You may need to reduce Max Memory to help the OS, but tuning Queries is going to be the best way to fight internal pressure.   Just adding Memory will NOT fix the problem.  You can't buy enough Memory to fix bad code, but you can give your best try.  Most of us would turn to Profiler to find these queries, but I will direct you to the SYS.DM_EXEC_QUERY_STATS DMV.  With just a few queries to that DMV, you can find the most expensive queries any way you want to slice and dice it.

Eric, @SQLPilot
Tweet

This time we are going to cover how to automate the checking of several policies on a folder in the Central Management Server.  

For this I am going to use a little help.   I like to use the Enterprise Policy Management Framework on Codeplex.   I think this is a great start to demo what is possible.  It also provides us with a great little PowerShell script.  Thanks @SQLGal for putting it up there to use.  All of the things below can be downloaded so you can do this in just a short time.  

This demo assumes you have SQL 2008, and SQL Server Reporting Services installed.  It also assumes that you have imported all of the policies in Part 1.

First download all of the files from codeplex, once unzipped you should have 3 folders.

• 0Setup Scripts
• 1Down Version Evaluation PS
• 2Reporting

STEP 1:
The first thing we need to do is to setup a database to collect all of this data.   The demo assumes you have a MDW database for the Management Data Warehouse.  You can use any database you like, but go ahead and create it now.
Open EPM_Create_Database_Objects.sql located in 0Setup Scripts.  We need to modify a few lines, but first we need to make sure we are in Command Mode.  In MSSMS click on Query -> SQLCMD Mode.

When that is done you will see 3 lines gray out.  Those are the lines you need to edit.
Set :SETVAR ServerName "WIN2008" to your server name :SETVAR ServerName "<YourServerName>"
Set :SETVAR ManagementDatabase "MDW" to your database name. :SETVAR ManagementDatabase "<Your Database Name>"

Once those two things are done, you can execute the script.  This will create the tables to accept the results from the PowerShell script.

STEP 2:
The next step is to put the PS1 file somewhere on the Central Management Server.   Again it does not matter where, just as long as the SQL Agent can get to it.
Create that folder, and also create a folder named Results in that same area.
Now that the PS1 file is in its final home, we need to edit just a few lines in it.   This file is just a text file, so open it in your favorite text editor.

 Here is the block we are going to touch.
# Declare variables to define the central warehouse
# in which to write the output, store the policies
$CentralManagementServer = "WIN2008"
$HistoryDatabase = "MDW"
# Define the location to write the results of the
# policy evaluation.  Delete any files in the directory.
$ResultDir = "e:\Results\"
$ResultDirDel = $ResultDir + "*.xml"
Remove-Item -Path $ResultDirDel
# End of variables

The only 3 lines we edit are...
$CentralManagementServer = "WIN2008"
$HistoryDatabase = "MDW"
$ResultDir = "e:\Results\"

Set those to the appropriate values and save the file.

STEP 3:
The next thing I like to do here is create a text file with the correct PowerShell call so I can keep that on the server also.  So create a new text file.  Below is a start, then you will need to modify for your information.
SL "<Path to PS1 File>"
.\EPM_EnterpriseEvaluation_3.0.0.ps1 -ConfigurationGroup "<Central Management Folder>" -PolicyCategoryFilter "Microsoft Best Practices: Maintenance" -EvalMode "Check"
I need to break on a bit of a tangent here.  Above we are putting in "Microsoft Best Practices: Maintenance".  Each Policy that we make has a Category.  You can make your own category, and then change this name.  If you imported the policies like I did in Part 1, then you can use this name.   Over time you will tweak the policies to you specific requirements, and then I would assume you use your own categories.

Okay, now just a few more things to do.  Now that we have the script in place, and the execution command laid out, we need to add the reporting piece.  This example was written for SQL 2008, so you will get some upgrade warnings if you open it in 2008R2.   This is fine, and it works just fine.
Open up BIDS and then open a project.  Navigate to \2Reporting\PolicyReports and open the project.
Once the project is open in BIDS you will have just a few areas to make modifications.
Shared Data Sources is the first.  Double click on PolicyDW.rds.  Now modify the connection string to the server that you created the database on.  It should be localhost, but if you made it a named instance make sure it's correct. 
The next step is to tell BIDS where to deploy (Install) this report.    In Solutions Explorer, Right Click on PolicyReports (All the way at the top), and select Properties.

 You will only need to modify the URL row here.  

Once that is done, we now need to deploy the project to the Reporting Server.
Click on Build -> Deploy PolicyReports

If all goes well you will have the project deployed to your report server.    Since we did not run the scripts yet, there will be no data to report on though.  

FINAL STEP:
The final few steps are going to be testing the script, and then installing that script as a SQL Agent Job to run every day, week, or month.
To test run this script you need to be in Power Shell.  The easiest way to get there is to go into MSSMS and right click on the server (Central Management Server), and then click Start PowerShell.   You will get a black dos looking prompt, but it will be PowerShell.
Past in the code from above;
SL "<Path to PS1 File>"
.\EPM_EnterpriseEvaluation_3.0.0.ps1 -ConfigurationGroup "<Central Management Folder>" -PolicyCategoryFilter "Microsoft Best Practices: Maintenance" -EvalMode "Check"

You should see a window like this appear, and the script will be running.

If you want to check and see if it worked, you can either open the Reports you deployed or just check the table.
SELECT * FROM Policy.PolicyHistoryDetail

Again I just want to point out that this is just one way to do this.   The Polices and PowerShell script will do all of the "Magic", all you need to do is create the Policies and the Reporting mechanism you want.

Eric, @SQLPilot
Tweet

How can we fix this?   Policy based management can help.   This is a take on the Policies in AD.  If you want something to be set on EVERY computer, you make a Policy; well the same is true in SQL now.   How do you know if a vender's install changes system settings?  Do you monitor every system setting on every sql instance every week, month?  My honest answer is I seriously doubt it.   At least I have not found a single customer that does in 3 years of working as a PFE.  I also did not do this when I was a DBA.

I'm going to take this post to talk about Policies, and how they can help you make some extra time.   Please don't think this is time you can put your feet up and sip a Latte, because there is plenty of work to do, trust me, I see it :)  

Just a bit if history... Back in my day (ha, just a few years ago) I also needed to do this "Enterprise Wide" stuff.   My job consisted of checking logs, backups, jobs, and moving objects from dev to stage, to prod.   The checking took 80% of my time as I had just around 100 servers that I was responsible for.   So what got dropped?  Well, I have to say it was the most fun stuff that did, TUNING.   During that move from dev to stage, we were supposed to review the code and try and tune it.   Re-write statements, add indexes, review the design, etc...   None of that got done, unless of course you had a performance problem, then that was done at that moment.   Usually with the boss standing behind you saying, "Is it up yet?".   

Why the history? The reason is how we all addressed the exact same issue.   I fixed it by writing LOTS of vbscripts trying to pull every detail of the first part of my job.  I did this for two reasons.  1. Who wants to do the same exact thing over and over and over? (NOT ME!). 2. I wanted to do the COOL stuff and make things faster and challenge myself.    I see these same kinds of scripts all over at my customers' sites now.   Here is the main problem.   There is usually one person who truly understands every little detail about them, and everyone else, just uses them.   No biggie, well tell that to them when that person leaves the company, and you are left holding scripts that need to have a new feature added, or a bug fixed.   Ok, ok, where is this leading?  To Policy Based management of course.

Policies, introduced in SQL 2008 are a way to take the custom scripts out and still get the work done.   All of the "Magic" happens in the policy itself, and then we can automate some things around it.  Don't worry, I'll explain it.  

Let's start very small.   Let's say we want to check the last backup date of each database.   Easy, we can use a policy.  This policy happens to be shipped with the install of SQL. So let's add that policy first, and then we will see how to use it.

Adding the Policy...

• Go to a central server, preferably the Server you built in the first post I wrote, "Central Management Server".
• Open Object Explorer, not Registered servers.
• Open Management -> Policy Management and right click on Policies    
• 
• Click Import Policy...
• Once that window opens, you will need to find the policies. Depending on your install path, this may be different, but here is mine. "C:\Program Files (x86)\Microsoft SQL Server\100\Tools\Policies\DatabaseEngine\1033"
• In that folder there are 50 different policies you can import. I recommend you import them all so you can play with them and see what is possible.
• Now click import and in a few moments you will have them on the server.

 Ok, now the policy is here, we can go and evaluate it locally.   You could do this on every 2008 server you have, but that's not really very powerful is it?   First we must walk, later we will run.  

If you open the policy's, you will find one called "Last Successful Backup Date".   Lets do a quick run or Evaluation of this.  Right click on the policy and then click "Evaluate"

 When you do, a new window will open and you will see the policy run.  It will show you the results of the policy as well as the details for each database it touched.

 As you can see, I don't backup my local laptop instances very often, but that is good to show you this.    Click on one of those "View" links under details and you will get a window like this...

 As you can see, this policy is looking at the API Property of LastBackupDate.  If that date is < NOW() - 24 Hours, or older than 24 hours old, the policy will fail.

 Ok, the cool stuff is out of the way, you have "Evaluated" your first policy, let's go take a look at what a policy looks like.  Close all of those windows and get back to MSSMS, and look at all of the polices.   Double click on Last Successful Backup Date.  This will open a new window so you can see it in detail.

 Click on the ... next to Check condition.

In this window, you will see the code that is to be evaluated.

Note: Facet: is grayed out.  This is grayed out when you review or change the policy; this is not grayed out when you create a new one.

 Ok, just to recap, we now know how to import a policy and evaluate it.   All of this is being done locally for now, I don't want this blog post to turn into a book.   There are several other directions I can take this, but I want to show you how this can be powerful, so we are going to evaluate this policy on all of the servers listed in your Central Management Server.    We are going to do this in a manual way this time, and I'll show you how to automate it as well as expand polices in my next post.

Alright, it's time to click over to Registered servers and look at the Central Management Server.

• Right click on your server name, ebackschSSD in my case, and select Evaluate Polices...

• In the new Window click the ... box and a new window will open.

• Click the Server Radio button, and then find the name of the server you just Imported the polices too. Click Ok
• You should now see a list of Polices that are on that instance. Scroll down and find the Last Successful Backup Date one and check the checkbox, and then click the Evaluate button.

• 5. You will now see a new window open and the policy will be evaluated for each database for each instance in that list.

 NOTE: You may notice the Server Names.  We need to use SQL 2008 to evaluate the policies, but we can evaluate them on older versions of SQL.

In my next post, I'll walk through how to automate this process with Power Shell to show the full potential of Policy Based Management.   I hope that you can see how this will help you better monitor your SQL Server Environment as a whole.    I encourage you to go and play with this.  Create some new Policies of your own.

 Eric, @SQLPilot
Tweet

Central Management Server

Why is this such a good thing?   Well it's easy, and it makes the DBA's life a bit easier.   Sounds like a Win-Win to me!   For some background, I see very large Enterprise customers most weeks.   These customers have DBA's and servers spread around the world.   What does this one little feature do?  Well it gives all DBA's a single list of ALL servers/instances.   Doesn't sound like a big deal?  Well if you have 300+ instances and 3000+ databases, it is.

This will let you build a structure around Dev, Test, Stage, QA, and Prod.  Separate those further if you like to App team, responsible DBA, or Location, etc...   Whatever works for your team.   

Now just a quick walk through how to do this.   First you need a SQL 2008 (2008 R2 Preferred) instance.   Here is the good part, it does not need any huge server behind it.   I do this on a Virtual Server most times.  Once the instance and client tools are installed,  we just need to open up Management Studio.

Once in MSSMS click on View -> Registered Servers.

After that is done you will see a new window show up there.  It will be named Registered Servers and have a few items in a list.  One of those items is a folder called Central Management Servers.  Right click on that and click Register Central Management Server...

That will open a new window to register a SQL instance.

In this window add the instance that you are connected to.   Yes, Register itself, you will see why later.

Once that is done, you should see a + sign next to Central Management Servers.  Click that and you will see your server name listed there.   Now we have a Central Management Server.  

Hold on we still have a few things to do.   I'm sorry to say that the Easy part is over.   Here comes the hard part....   You had your team have to come up with the structure of this folder layout.     I guess this means we have to have a meeting, Yes it usually does, but it can be brief.   I would recommend starting with something very simple.   Dev, Test, Stage/QA, and Prod.    Add a few servers to each one and then show what it can do.

First here is what it might look like.

 Now that we have done all of that work, what did we get out of it?   Well, go back to your desktop and open SQL 2008 R2 Management Studio.   Do the exact same steps as above.

• View -> Registered Servers
• Right click on Central Management Servers and select Register Server.
• This time, you need to add the name of that instance that you just used above.
• If you followed all of the above steps, you should be able to see all of the folders and instances that you registered previously.

This is the first cool thing here.   If all of the DBA's add this to their local install of Management Studio, then you ALL will have the exact same list of servers.   If one of you add a new one, then you all get to see that latest and greatest list.

There are a few things that you can do here, but this basically is a lead into Enterprise Management for SQL Server.   Best of all, this is included with SQL, so no 3rd party products to buy.

What else can I do?

• If you right click on your Central Management Server, you will see that you can Open "New Query" and "Evaluate Policies"
• "New Query" this will let you run a query on all of the instance below. The query must be valid on all instances (SQL 2000, 2005, 2008, 2008R2) Try something quick like "Select @@version"
• "Evaluate Policies" this one I won't get into detail about as it will be my next Blog Post, but this is one way to Evaluate Policies across the entire environment. This is how Policy Management for SQL starts, but it gets more powerful

Just a few things to tie it all up.  

• SQL 2008R2 is preferred for this Central Server. If your building it new, why install an older version?
• SQL 2008R2 Client is preferred on all of the clients.
• All servers registered must be NT Authenticated. You will notice there is not an option for SQL User.
• If one of you change this list on your PC, it will change the list for EVERYONE!