Geek of All Trades: Patching Dormant VMs
Using the Microsoft Offline Virtual Machine Servicing Tool to patch dormant VMs is a good way to ensure they’re ready to go live, and avoid any problems with out-of-date patches.
By Greg Shields
Do you hate patching? I certainly do. Patching has always seemed like a thankless task, made even more menial by the fact that there’s no business value in doing it. Well, at least no business value other than the knowledge that you’re probably avoiding some unknown catastrophe down the road.
No, patching has long been the grunt work of the IT world. A major source of late nights and after-hours, “Sorry, honey, I won’t be making it home for dinner,” calls, patching and patch management is the activity we all love to hate.
These days, however, patch management isn’t nearly as bad as it used to be. Not long ago, keeping patches straight required a massive spreadsheet, linking each patch to its “MS” number and “q” number, its Microsoft criticality with our corporate priority. Keeping track of which patches superseded other patches consumed at least half a day every month.
Much of that changed with the release of Windows Server Update Services (WSUS). This simple but fantastic timesaving tool took most of the manual labor out of patching. Combining WSUS with a little scripting, you could even instruct it to patch a computer immediately without waiting for the next scheduled cycle. (I still have that script. Stop by concentratedtech.com if you’d like a copy.)
The one limitation with WSUS is that its automated patching mechanisms only worked when the server or desktop you needed to patch was actually turned on. Computers that for one reason or another had to be turned off were a problem for WSUS. This required a patch cycle to occur immediately after the machine was powered back on the next morning, or extra work for the administrator in finding and powering it on the night before.
By now, we all know this limitation with WSUS isn’t that big a deal. If users leave their machines off during the evening patch cycle, they’ll get a nice balloon notice the next morning after they power on and the patches start installing. Even today, servers generally stay on all the time. This means they’re always up and available to receive instructions from their WSUS patching overlord.
Changing the Mix
This pleasant and static environment changes once again as our datacenters jump to virtualization. Once virtualized, our servers are still likely to remain powered on constantly. Those virtual servers have to come from somewhere, though. For the vast majority of us, that “somewhere” is through the cloning of a server template.
Server templates are great because they help us quickly spin up a new server and bring it online with minimal effort. They also ensure that every server starts its life from the same core configuration. Server templates also have a darker side as well. While these computers are great for birthing a new server, our templates themselves aren’t intended to have lives of their own.
A server template exists as a VHD file on a file share somewhere. Powered off, this file is effectively dormant. It’s no different than a really big Word file or Excel spreadsheet. Powered on, that dormant file becomes a fully functional server, one that can process regular workloads, or the nefarious workloads of today’s malware and other exploits.
In short, if those dormant templates aren’t patched, they could become the source of a brand new malware outbreak—all because they were simply powered on.
Got your attention? Good, because that’s the central theme to the Microsoft Offline Virtual Machine Servicing Tool, currently in version 2.1. This free solution accelerator installs a set of automations intended to keep your otherwise-dormant virtual machine (VM) templates up-to-date with WSUS.
.png "Figure 1 The three major components of offline VM servicing")
Figure 1 The three major components of offline VM servicing.
To get the gist of how this works, check out Figure 1. You can see how the server that hosts System Center Virtual Machine Manager (VMM) interacts with one of its Library shares, a Hyper-V host, and your software update management server. That update management server can be a WSUS server or an available System Center Configuration Manager (SCCM) server. For either, the processes are about the same.
The Offline VM Servicing Tool uses what are called servicing jobs to manage deploying updates to VMs in the VMM library. These servicing jobs are essentially tasks handled by the Windows Task Scheduler that kick off at predetermined times.
When a service job is ready to activate, it starts by first “waking” the VM from its template location. The waking process involves deploying the VM to a Hyper-V host and powering it on. Once that VM is powered on, the VM’s internally configured Windows Update Agent (WUA) is instructed to begin a software update cycle.
The VM’s WUA configuration is set the same way you’re used to setting it for all your computers. This can be through Group Policy application or by manually setting the configuration within the VM. You have to set all the typical WSUS configurations for this process to work, such as the update service location, any WSUS computer groups, as well as other specific characteristics as defined by your security policy.
Once you’ve successfully updated the VM, the servicing job powers it down and returns it to the library. This automated process ensures that your VMs—as well as the rest of your infrastructure—are updated with the correct patches. The Offline VM Servicing Tool does not add any of its own update management configurations, leaning instead on the existing settings you’ve already configured for WSUS or SCCM.
Actually, getting the Offline VM Servicing Tool installed requires a few steps that aren’t immediately obvious if you haven’t read its associated Solution Accelerator guide. While at version 2.1, this tool still feels like an early release. Installing it requires a console download from the Microsoft Web site. Getting it to run requires the extra step of downloading and copying PSExec’s binaries—both psexec.exe and pdh.dll—to the tool’s bin folder, found in C:\Program Files\Microsoft Offline Virtual Machine Servicing Tool\bin. It also requires the Windows PowerShell execution policy to be configured for RemoteSigned.
.png "Figure 2 The Offline VM Servicing Tool console")
Figure 2 The Offline VM Servicing Tool console.
Once installed, its console (shown in Figure 2) really only identifies Virtual Machine Groups for determining which machines get patched when. It also identifies Servicing Jobs, which contain the characteristics of your update tasks. The servicing tool also only works with VMs contained within your VMM Library. This means any powered-down VMs that have already been deployed to a Hyper-V host won’t work with the tool.
It will work with VMs that aren’t specifically templates, such as any hot spare VMs you have ready to bring online after a failure or when you need additional servers. In these cases, the tool suggests using a secondary NIC on hot spare servers along with deploying to an isolated network. This helps ensure that the hot spare doesn’t accidentally become operational when you only mean to apply current patches.
The Microsoft Offline Virtual Machine Servicing Tool may not be a complete solution for all your offline patching needs, but its inexpensive price and limited capabilities will help in those situations where you just want to keep a few dormant VMs up-to-date. Considering the pain of doing this manually, combined with the impact of a missed patch if you don’t, you can see how critically important it is to keep tabs on all your dormant, but potentially dangerous, VMs.
.jpg "Greg Shields")
Greg Shields*, MVP, is a partner at Concentrated Technology. Get more of Shields’ Jack-of-all-Trades tips and tricks at ConcentratedTech.com.*