I created a number of VMs on the systems, and configured replication, including extended replication for a couple of the VMs, as per the picture below:

Like I said earlier, what a mess of replication I’ve created! 
How could I possibly keep track of it all, as well as effectively monitor replication?

Truth be told I created this mess so I could create a report!  I was inspired somewhat by on of Ben Armstrong’s post about his home configuration. 
Ben created a nice little PowerShell one-liner that created a simple, easy to read report detailing replication status on a single host.

get-vm | Get-VMReplication | select name, replicationmode, replicationhealth,  @{Expression={"{0:N0}" -f ((get-date)-($_.lastreplicationtime)).TotalMinutes};Label="Delta (min)"}

…and the output looked like this:

It’s a nice little bit of code…Ben did some things with time to make the replication delta time appear in minutes – very cool.

I wanted to shoot for something a little more ambitious.  I wanted to report on multiple servers (to help with monitoring  remote branch / store locations), as well as make it sortable.  Ben’s output didn’t show any of the host names or the assigned replication interval, and (as is the way with PowerShell) it looked sort of old school with the output sitting inside the PowerShell Window.

I tweaked Ben’s one-liner a tiny bit.  First I created a text file on one of the hosts (c:\Serverlist.txt) with the names of my Hyper-V hosts, so I could loop through them:

johnkel-hpa.replicatest.com
johnkel-hpb.replicatest.com
johnkel-hpc.replicatest.com
johnkel-hpd.replicatest.com

I know that I could have simply asked AD for all the Hyper-V hosts it knows about, but I wanted to create a simple example (and no, I don’t own replicatest.com – I’m not running this on the Interwebs … just using the name on my coffee table!).

Then add some missing fields and pipe it to one of my favorite PowerShell cmdlets Out-Gridview:

get-content .\serverlist.txt | foreach-object {get-vm -computername $_ } | where {$_.replicationstate -notmatch "Disabled"} | get-VMreplication | select primaryserver, replicaserver, name, replicationmode, State, replicationhealth, @{Expression={"{0:0.0}" -f ($_.FrequencySec / 60)};Label="Target Freq (min)"}, @{Expression={"{0:N0}" -f ((get-date)-($_.lastreplicationtime)).TotalMinutes};Label="Delta (min)"} | out-gridview

The output is a sortable, filterable GUI (via Out-Gridview) with all the critical info I think I would need to monitor the status of Hyper-V Replica on my servers:

PowerShell is awesome… but I must confess that I don’t use it everyday, so I have to poach great examples and fiddle a lot .

Still, what’s great about the basics of the one-liner that I snagged from Ben is that it can be used in any number of ways – it could be updated to to create a simple exception report for VMs with an “Error” for replication State, to find VMs not configured for replication…whatever.

The great thing is that this “one-liner” let me easily see (and unwind) the multi-server replication I had created!

-John

Here in the U.S. it is now the post-Thanksgiving, Christmas season…what does that mean?  SHOPPING!

Remember that Thanksgiving has a long tradition here in the U.S. of supporting retail sales – that’s why the date changed back in the last Century...to give stores more time to sell before Christmas!

Reliance on systems for price lookups, tax calculations, and credit card processing are critical, time sensitive functions that cannot be offline for long during this season, or revenue will be lost!  Modern retail store locations rely on at least one server and often times multiple servers to provide these functions.

The struggle is that margins are thin in retail.  Any cost must be weighted against the value it provides…so how much is the right amount to spend to protect time sensitive transaction processing? 

Retailers have struggled with reducing the cost of store infrastructure for many years, and virtualization has helped by allowing multiple servers to be combined onto fewer physical servers.  Really every industry with smaller, remote branch locations has similar challenges - banking, education, food service, manufacturing, distribution. 

Often the next logical step in protecting processing is to introduce VM High Availability (HA) which has traditionally been delivered through clustering (with Hyper-V or VMware).  The problem with HA is that it usually requires storage that can be shared between multiple physical servers…like a SAN, and that type of storage is expensive for a retail store!

It doesn’t matter what virtualization technology you use…shared storage and traditional clustering add cost and complexity to a deployment.

I’m not knocking SANs, shared storage, or clustering – I use all of them and REALLY like them, it’s just a struggle to cost justify shared storage and clustering for say 1000 sandwich shops or 100 schools in a school district.  Is there enough value in the added expense?

Economics of Downtime

What do retailers (and organizations with branch offices) want from clustering?  They want availability.  They want to protect processing from hours or days of downtime when there’s a hardware failure.  They may also want the flexibility to perform planned hardware maintenance without unplanned downtime.

Clustering offers planned migration of VMs between physical servers without unplanned downtime as well as the automated restart of VMs when there’s a hardware failure (on another host), but that expensive shared storage is still a single point of failure.  Yes you can throw more money at it to make it redundant, but that’s not the solution.

You can’t eliminate all risk and downtime – but you can reduce risks.  The big question is how much down time could a store / branch tolerate?

What if you could move VMs between hosts with little or no downtime  - WITHOUT shared storage (you know you can already with Hyper-V, right?). 

Hyper-V Replica allows you to shutdown a running VM, and fail over to it’s replica with only a tiny amount of down time.  That’s great for hardware related maintenance if you don’t need to Live Migrate the VM

What if you could replicate VMs to alternate hardware that could be started quickly in the event of a hardware failure WITHOUT shared storage?   When a physical server at a branch goes down, what if you could start the same VMs on another host at that location in a matter of minutes without an investment in shared storage?  THAT’S WHAT YOU CAN DO WITH HYPER-V REPLICA!

Remember that Hyper-V Replica is included in Windows Server 2012 / Windows Server 2012 R2 and the associated versions of Hyper-V Server (which is a completely free download!).  I’ll say it again, ASNYCRONOUS REPLICATION OF VMS ON HYPER-V IS FREE!   Your VMs need to be properly licensed (Windows / Linux / applications / other), but this awesome feature is built in to a completely free product!

What’s Different From a Cluster?

The big differences are that you don’t have any single set of storage that needs to be connect to multiple hosts, and you don’t have cluster server watching to make sure that the VMs are available (to fail a VM from one box to another when there is a hardware failure).   Because there is no cluster service using Hyper-V Replica, the VMs do not “out of the box” automatically fail over to the alternate physical host as they might with a cluster.  You can certainly automate any failover if you want that to happen, but do you really need the VMs to do that?  If you did, then you would deploy a cluster, right? 

If you (the administrator) need to poke around and see what’s up first (as you typically would if you were performing planned maintenance or had been notified of a failure), is it really that big a deal for you to “right click” and “Fail Over” the VMs?
(Note that there are a bunch of ways to failover…we’ll get to Hyper-V Recover Manager later this Winter – I hope, be patient).

Perhaps you could add a button to the store managers desktop that did the failover for them when they have an issue (you know that would be a snap to automate with PowerShell).

Back to the big question - how much down time could a store / branch tolerate?
If your branch VMs were already on another server and ready to start up, is that good enough to get the operation back on line?

What about protecting data and services when the whole site goes down?  There’s something that’s harder (and move expensive) to do with a cluster or VMware (ESX doesn’t come with replication).

Remember that 2012 R2 includes a tertiary replica… you can extend the replica to a third Hyper-V host that could be in back at a central data center!

OK, so that’s one of the values offered by Hyper-V Replica – rapid recovery / migration of branch services without the cost or complexity of clustering.

Can you do that with what’s included with ESXi?  I think not!

I’ll get into some PowerShell tricks in my next post.

-John

I’m using my coffee table once again (like I did with my USB 3.0 Tiered Storage SAN), and I’ve created a Hyper-V test infrastructure based on a stack of “heritage” laptops (not new, but Hyper-V capable). 

I have four laptops for now – one is a DC and all of them are running Windows Server 2012 R2 with Hyper-V enabled, connected by a single, GigE switch.

I’ve added a couple of VMs to each, and configured them to replicate from their primary host to at least one other host, and to complicate things I’ve Live Migrated some of the VMs from their initial host (HPA) to another host (HPD).

Hopefully the naming conventions make a little sense (they are there mostly for me to keep things straight) – the four host names each end with a different letter (A-D) and the VMs created on each host were named based on the host suffix.  As I mentioned earlier, the VMs initially created on HPA were setup to replicate and then later Live Migrated to another host…why the names of the VMs on HPD don’t match the VMs homed there.

Wow – what a mess of replication I’ve created! 

How could I possibly keep track of it all, as well as effectively monitor replication?

I’ll leave that for subsequent posts.

-John

Many of these posts don’t remind you of some of the basics…like how to easily view / monitor replication. 

Within Hyper-V Manager, you likely already see the “Status” of VMs – that column shows activity, but you can also view the overall “Replication Health” – something that isn’t displayed by default.

You will want to update Hyper-V Manager to see “Replication Health”.  Within Hyper-V Manager, right click on the column headers in the center window, and select “Add/Remove Columns” – then you’ll be able to add “Replication Health” to the console, as well as reorder the columns.

Here’s what my Hyper-V Manager looks like after changing the displayed columns:

This simple change to Hyper-V Manager makes all the difference for administrators who want to view replication health.

Don’t worry about the VMs showing “Critical” – I caused that on purpose so we could find it and fix it later!

Next I’ll go into some details on my Hyper-V Replica test setup, and then later show you some ideas I have on monitoring larger deployments of Hyper-V Replica.

-John

I like to think that one of the “target” scenarios for Hyper-V Replica was disaster avoidance - rather than simply disaster recovery (DR) – the ability to move a VM or service far away from an anticipated threat BEFORE it causes an outage.  The perfect example being natural disasters like hurricanes…with hurricane Sandy being a great example of how the built-in capabilities of Hyper-V Replica can keep things running:  Hyper-V Replica and Hurricane Sandy.

Hurricanes don’t hit our data centers and work locations every day.  More commonly other “disasters” strike… things like drive failures, water leaks, or well meaning janitors who trip over a power cord. 

Have you thought about Hyper-V replica and how it could help avoid or limit outages in other scenarios?  

• In branch offices / retail store locations (for example) have you avoided deploying Hyper-V or VMware clusters to improve availability because of the high-cost and complexity of shared storage?  You can use Hyper-V Replica without shared storage to provide local failover, as well as centralized backup!
• What about using Hyper-V Replica for your development environment…why not turn it on to get backup copies of VMs with multiple, VSS consistent recovery points without a lot of muss or fuss?

There are lots of great scenarios enabled by Hyper-V Replica – more than I could possibly conceive.

I’m going to spend some time walking through various scenarios over the next few posts, as well as how I might "automate” things like monitoring via PowerShell.  I’m not an expert on Hyper-V Replica, but I see it’s value, and have received LOTS of questions about it from people who are beginning to understand it’s power. 

If you have any suggestions, comments, or questions please don’t hesitate to ping me.

-John

Since I posted about “Storage Rig” and the walk through of how to setup tiered storage, I found that the USB 3.0 sticks I had been using were VERY slow on writes, and not well suited for playing around with tiered storage.

I swapped out the two USB 3.0 sticks for Samsung 840 Pro Series SSDs in the same USB 3.0 trays I used for the various spinning disks I had. 
These SSDs are blazingly fast, and can (Based on their specs) support up to 90,000 write IOs per second!

I’ve been asked by a number of people about performance testing with my setup.  Let me first say that all the testing I have done is less than scientific…it’s been simply playing around with inexpensive parts I had laying around and pieces that I spent less than $100 in total on…except of course for the Samsung SSDs.  To put some of the performance constraints on the table, lets review some of the theoretical limits of my setup:

• Lenovo W530 laptop with onboard USB 3.0 – fast laptop with internal SSDs
• USB 3.0  - theoretical maximum throughput of 5 Gbit/s
• SATA 3.? – theoretical maximum throughput per drive of 6 Gbit/s

Using a single USB 3.0 connection to my host, and a single USB 3.0 hub, it would appear that my gating factor is USB 3.0 (5 Gbit/s).

I created my virtual disk and volume, using all the same drives with the new SSDs.
For giggles, I copied a bunch of exported VMs (large VHD / VHDX files – about 16 GB or so) from a local SSD to the new “tiered” volume using Explorer (you know, right-click on a folder in D: to copy, and then right click on my new volume to paste).

So, what did I see in my stunningly unscientific testing?  I saw a VERY FAST COPY OF A LOT OF BIG FILES!

Did you see that?  666 MB/s…that’s 666 MegaBYTES / second!

Wait, wait, wait…so there are 8 bits in a byte…so 666 * 8 is 5238 mb (megabits)…maybe divide by 1024 and I get 5.2 Gb/s?

That’s faster than USB 3.0?  How can that be?  Well, in reality  it can’t be…it’s just what’s reported intermittently in that silly copy window when you use File Explorer to copy.  Were the files copied REALLY fast, certainly…but using my entirely unscientific testing methodology doesn’t let me measure accurately! 

Yeah, for accurate testing, you really can’t use File Explorer for a bunch of reasons.  I could have certainly used robocopy…that would have been good, but the copy window above was good enough for me. I’ve never seen performance like that on any of the expensive SAN gear I’ve used…so to see it on a laptop with a bunch of old drives and a couple of the latest Samsung SSDs is just stunning! 

The punch line here is that WINDOWS SERVER 2012 R2 is A HIGH PERFORMANCE STORAGE CONTROLLER… even if it is just a laptop on your coffee table.

Does my systems actually performance matter?  Not really, because YOU SHOULD NEVER USE USB ATTACHED STORAGE LIKE THIS ON A SERVER!

What I’ve done here is just silly.  There are all sorts of reasons why what I did here isn’t meant for the real world, like I bought really inexpensive enclosures (the drives flop around in them) and USB means I have all softs of power problems (these enclosures are bus powered) from having to spin up drives one at a time to seeing them sometimes go to sleep.  To do this right, you should use SAS or SATA attached drives (really SAS).  To find tested and certified subsystems, just go to the Windows Server Catalog site – that’s where you can see what’s been tested and can be used in a scale out file server or Hyper-V cluster.

With storage tiering baked into Windows Server 2012 R2 you can take off the shelf parts and achieve astounding performance with a tiny investment of time and money. 

-John

In my last post, I showed how I created a Storage Pool. 

Now I’ll walk though defining a virtual disk and a volume using the available wizards in Server Manager.

Start by going to Server Manager and opening up the File and Storage Services, and selecting Storage Pools on the left side of the screen.

In the VIRTUAL DISK window, select TASKS and click on New Virtual Disk:

This opens the New Virtual Disk Wizard – you can read and then hit Next:

A virtual disk is created from an existing storage pool…I have only one, so hitting Next is a good option:

The next screen allows you to assign a name to your virtual disk, as well as let Windows know that you would like create storage tiers on the virtual disk (use those two separate media types defined in my previous blog post to optimize performance):

The storage layout must also be selected.  With my storage, I was allowed to select either a “Simple” striped layout or a mirror (simple in this case has no fault tolerance…but I don’t need that right now for my testing) :

In some configurations, Storage Spaces allows you to enable “thin provisioning” of volumes…I don’t have that option here, based on my setup, but it’s a wonderful capability:

I defined my virtual disk to be the maximum size it could be, based on the available capacity of my storage pool:

The Wizard provides the opportunity to review the virtual disk options, and then create the disks:

The progress, and lastly the results are displayed once the virtual disk is created (note the check box to start the New Volume Wizard at the bottom):

Now the new virtual disk has been created, and is visible within the storage management interface:

As noted, the New Volume Wizard will kickoff by default after a new virtual disk is created:

…and our new virtual disk is there on which to create our volume:

Just like with a physical disk, I can create multiple volumes (if desired) on a virtual disk, up to the capacity of the virtual disk, as are the options for assigning a drive letter or mounting the volume as a folder, picking a file system, and assigning a volume label:

One option you may not have run into previously, is the option to enable Data Deduplication.  What’s new in 2012 R2 is the ability to specify the “type” of depuplication – dedup for general purpose file server services, or for hosting VMs for VDI:

Once again you’ll see a summary screen detailing the options for the volume to be created, followed by progress and completion status:

The new volume hosted on tiered storage is now available and visible in all those common places we might want to see it:

That’s it!  That’s all that I needed to do to create and use my high-performance, coffee table-based storage sub-system!

I’m going to take a break now for a couple of days, and see what questions and comments come in.

Let me know what you think about this tiered storage walk-through, as well as what else you might want to see me test.

-John

Now I would like to show you how I created my high-performance storage pool with the remaining, unallocated memory sticks and external drives.

The first thing to note, is that you do not use the traditional “Disk Management” interface you may already be familiar with to creating storage pools / virtual disks, although the underlying physical disk: 

Instead, you use the 2012 (R2) Server Manager user interface – and select “File and Storage Services”, which will bring up a comprehensive set of storage management screens within Server Manager:

Notice that you can see the same defined volumes as well as attached disks in the user interface, by navigating through the various options on the left side of the screen:

To create my storage pool, I navigate to Storage Pools, and then can see the available physical disks:

The information included in the “PHYSICAL DISKS” window, is extremely valuable, as it includes key information like the Bus (all my disk is USB attached), and the Media Type (listed as Unknown, since USB doesn't apparently report that or the spindle speed).

It’s important for Windows Server 2012 R2 to know the medial type, so it can properly provision storage tiers (I’ll show you how to set media type in bit, once we create our pool.

To create a new storage pool, go to TASKS in the top right, and select New Storage Pool, which will open a New Storage Pool Wizard:

You enter your new Storage Pool Name, and hit Next:

You select the available locally attached disks to add to the pool and hit Next again:

The Wizard displays a summary, and lets you confirm the creation of your pool:

Depending upon the number of disk and the performance of your gear, it could take some seconds to complete the creation of your pool:

You will now see your newly created Storage Pool, and no longer see the unassigned disks:

Interestingly, you will no longer see the disks assigned to the new pool in the traditional Storage Management interface:

This really highlights the point that you should perform your storage management tasks from within Server Manager!

Remember, all your storage is right there in Server Manager – I showed you the volumes before in Server Manager – that hasn’t really changed, as I haven’t created a virtual disk from my Storage Pool and subsequently a volume.

Let’s go back to Server Manager and click on our Storage Pool:

Remember I mentioned that the “Media Type” was not set (displayed as “Unknown”) – we need to update that for each drive to properly utilize Tiering.

We just have to touch each physical disk in our pool with a little PowerShell, and we are all set:

Note that you can’t update the media type until after the disks have been added to a pool (why I waited until after the storage pool was created).

Now the disks have the properly set Media Type to proceed:

The storage pool is created!

In my next post, I’ll create a virtual disk and a volume.

-John

My goals for this effort are:

1. Provide a walk through of Storage Spaces – I’m stunned how many admins haven’t had a chance to look at it or use it
2. Show how to integrate SSDs (or my case fast memory sticks) into a storage pool
3. Demonstrate the potential performance gain of using a mix of storage technologies, with the tools included in Windows

Somewhat Unscientific Methodology

I don’t have an unlimited budget, or unlimited time, so my initial testing was somewhat limited.

I’m using a laptop with USB 3.0 that has two physical (SSD) drives in it, with the additional external drives attached via USB 3.0.  For fun I’ve included a Powershell listing of the drives (using get-physicaldisk):

Disk2 and Disk3 are 32GB USB 3.0 flash sticks, with the other disk (Disk4-7) being USB 3.0 attached SATA disk I salvaged from old laptops and what not.

For testing, the first step was to create a “baseline” environment to act as a control.  I configured one of the four spinning disks (Disk4) as a standalone to perform the same copy tests I would use on the Storage Spaces volumes.

 Now I’ll move on to how I configured the volume in Storage Spaces.

-John

My goal was to create a high-performance storage sub-system which I could use to experiment with Storage Space and the new capabilities in Windows Server 2012 R2, but do it on the cheep.  USB 3.0 supports extremely fast data transfer rates, and I just upgraded to a laptop with USB 3.0, so USB 3.0 attached storage seemed liked a great option (plus I already have two USB 3.0 32GB flash drives for use with Windows To Go that I could repurpose as my SSD tier).

I picked up a powered 7-port USB 3.0 hub, and a bunch of USB 3.0 SATA disk enclosures (bus powered), and found some salvaged SATA disks from old laptops (250GB-320GB), and plugged it all together.

I loaded Windows Server 2012 R2 on my laptop (which already had two drives installed) and added some drivers.

Once everything was plugged in and spinning, I had a bunch of disks, ready to use!

Important Safety Tips for Using USB!

Too Many Drives Starting at Once is a BAD THING

There’s a reason servers spin up one or two drives at a time…it’s because STARTING a spinning hard drive TAKES MORE POWER than keeping a drive in motion!  To get my storage system working properly, I have to plug each drive enclosure in to the hub one at a time, so I don’t draw too much current all at once…sort of like Apollo 13.’  If you want to try something like this yourself, do not use bus powered drive enclosures to limit the power draw from the USB hub.  My hub has a 1.0 Amp external supply, but I still have to manage drive startup manually with all of the bus-powered enclosures.

USB Manages Power

The drives on my USB powered storage spin town pretty regularly, based on all the great power management “features” backed into USB and Windows.  That’s not optimal for a production (or perhaps even a test) storage rig, but it is what it is.  Be aware that this configuration is not meant to be used for real life storage applications, but simply for me to show you how to use Storage Spaces and configure Tiered Storage.

Next up, I’ll cover my goals for testing.

-John