Optimizing and Troubleshooting Hyper-V Networking

https://shop.oreilly.com/product/0790145382924.do

Optimizing and Troubleshooting Hyper-V Storage

https://shop.oreilly.com/product/0790145383068.do

I was honored to write few sections and tech review both books for Mitch. Enjoy!

Download the following MP and Import into your System Center 2012 Operations Manager infrastructure. You will need to edit the MP as described in Chapter 5 and 6 of “Microsoft Office 365 Administration Inside Out”.

It is important to understand that these synthetic transactions only monitor specific entities and relying on them to determine the state of your O365 tenant can lead to false positives. Let’s use Microsoft Exchange as an example. Most Microsoft Exchange servers store mailboxes in more than one mail database, known as the EDB file. There are a variety of reasons for administrators doing this, but the point here is that monitoring a mailbox’s availability only reflects its mail database availability. In O365, mailboxes may be spread across a number of mail databases. This means we may be incorrectly stating that email is running, while only some mailboxes are available.

Microsoft Office 365 distributes mailboxes across numerous mail databases, abstracting this application layer from the user and customers. Monitoring this O365 mailbox will only validate that the mail database it resides on is available. This begs the question; how do we monitor O365 more accurately?

Office 365 has a component called the System Health Dashboard (SHD), which is a matrix of health states of various O365 workloads over the last 7 days. As you can see, O365 has 8 Exchange Online workloads that are exposed in this dashboard and need to be monitored along numerous others services and workloads.

O365 provides the following 7 services and 35 workloads:

• Exchange Online – Email and Calendar access, E-Mail timely delivery, FOPE Administration, FOPE Encryption, FOPE Quarantine, Management and Provisioning, Sign-In and Voice mail
• Identity Service – Administration and Sign-In
• Lync Online – All Features, Audio and Video, Dial-In Conferencing, Federation, Instant Messaging, Management and Provisioning, Mobility, Online Meetings, Presence, Sign-In
• Office 365 Portal – Administration and Portal
• Office Subscription – Licensing and Renewal, Network Availability and Office Professional Plus Download
• Rights Management Service
• SharePoint Online – Access Services, Custom Solutions and Workflows, InfoPath Online, Office Web Apps, Project Online, Provisioning, Search, SharePoint Features, SP Designer and Tenant Admin

The O365 Service Health Dashboard (SHD) is the authoritative source for monitoring an O365 tenant. This means we need to query the SHD with SCOM to determine the state of the services and workloads we are interested in alerting on.

The concepts behind this entails 3 different parts:

1. Run a script from a SCOM Watcher Node that queries the SHD
2. Have the script login to the O365 Admin Portal (https://login.microsoftonline.com)
3. Have the script read the servicestatus.aspx page and determine which one of the 9 values exists for the workload being monitored
4. Have the script write the appropriate event log entries and update monitor status
5. O365 custom MP rules pick up event entries and generate alerts accordingly

We have been leveraging PowerShell and have made progress, but not successfully logged-in due to, what we identified as a Java component requirement.

=================================

$r = Invoke-WebRequest https://portal.microsoftonline.com/ServiceStatus/ServiceStatus.aspx -SessionVariable O365

$O365

$form = $r.Forms[0]

$form | Format-List

$form.Fields["cred_userid_inputtext"] = "username@yourdomain.onmicrosoft.com"

$form.Fields["cred_password_inputtext"] = "password"

$r = Invoke-WebRequest -Uri ("https://portal.microsoftonline.com/ServiceStatus/ServiceStatus.aspx" + $form.Action) -WebSession $O365 -Method POST -Body $form.Fields

$r.StatusDescription

==================================

Next we need to query the E-Mail and calendar access section of the aspx page. From this section of the page, we can retrieve the current state of the workload.

We are retrieving the first column because it is the only state we need. O365 keeps 30 days of history for the tenant service monitoring, but SCOM can maintain months to years’ of data.

As of this posting, we have completed the following PowerShell commands to accomplish this. This script is a work in progress because we are at an impasse attempting to move beyond the login page.

==================================

$EMailServiceName="E-Mail and calendar access"

$HTML = gc '.\Service health.htm'

$section=$false

$result=""

foreach($line in $HTML)

{

   if($line -match "ServiceNames" -and $line.Contains($EMailServiceName)){$Section=$true}

   if($section -and $line.Contains("TodayCol") -and $line -match "title=`"([^`"]*)`""){$result=$matches[1]}

   if($line -match "SubServiceNames" -and -not $line.Contains($EMailServiceName)){$Section=$false}

}

write-host "Service:" "'"$EMailServiceName"'" "Curent Status is:" $result

Finally, once the value is retrieved, we can write to the Application Event Log like this:

write-eventlog -logname Application -source O365MP -eventID 301 -entrytype Information -message "O365 message" -category 1 -rawdata 10,20

==================================

There are 9 states an O365 Service/workload can be in. This table identifies them and how we could reflect them in SCOM:

There are many pages online that guide you through creating a monitor in SCOM. Here are a few we recommend. While they may be written for SCOM 2007, they are applicable for SCOM 2012.

How to Create a Monitor

How to create a monitor based on a script

Create a Script-Based Unit Monitor in OpsMgr2007 via the GUI

Creating a SCOM monitor for an average value

Once this process is complete and working successfully, select a different workload to retrieve the state and repeat for each workload you are interested in monitoring.

This posting is provided "AS IS" with no warranties, and confers no rights. Use of included utilities is subject to the terms specified at https://www.microsoft.com/info/cpyright.htm

Windows Server 2012 Hyper-V made it easier by including that option in VM’s Memory settings.

When you start a VM with Dynamic Memory enabled, guest Operating System running inside the VM only has the knowledge of RAM that has been allocated to it at the moment and this is normally equal or around the value of Startup RAM. You can verify this by looking at Installed Memory (RAM) value on the System’s page or Installed Physical Memory (RAM) value on the System Summary page when you launch msinfo32.exe

This number changes in an upward fashion until the next reboot, meaning that as VM’s memory pressure increases, Hyper-V will allocate more RAM to that VM increasing the above mentioned value. Now, for any reasons, if the VM’s memory pressure drops and Hyper-V balloons out part of that RAM from the VM, the value that guest OS has as the Installed RAM won’t decrease. Thus, User Interface and any WMI/PowerShell queries will show the maximum amount of RAM that has been allocated to that VM since the last reboot.

Here’s a PowerShell sample code that shows the RAM formatted in GB;

"{0:N0}" -f ((Get-WmiObject -Query "select * from WIN32_ComputerSystem").TotalPhysicalmemory/1GB)

Now, why do we use this?

There are many cases that applications or installer packages are looking for a minimum amount of RAM as a prerequisite. Some applications might not even launch unless the available RAM query return a satisfactory value. If you’ve ever tried to install Microsoft System Center 2012 components in a VM with Dynamic Memory enabled, you’ve probably seen the available RAM warning message in the prerequisite analyzer wizard. Not fulfilling that requirement won’t stop you from installing any System Center 2012 components though.

Startup RAM value will allow you to allocate that much RAM to the VM upon the start of VM and let guest OS sets the Installed RAM value accordingly. This RAM will be locked to the VM while it’s booting and the extra amount of RAM that isn’t needed by the VM will be available to Hyper-V as soon as VM’s guest OS and Hyper-V Integration Services within the guest OS are up and running. In most cases the actual physical RAM allocated to the VM could go down to the value of minimum RAM if there’s no memory pressure. This all depends on how fast your VM boots.

Off course, you can set the Startup RAM to same value as Maximum RAM for multiple VMs but you’ll need to be careful if the Maximum RAM’s too high and those VMs could start at the same time, some of them might not be able to boot, if the sum of the Maximum RAM values exceeds the total physical RAM available on the Hyper-V server. And yes, you can have the Startup delay action for those VMs and address this concern.

In summary, Dynamic Memory Startup RAM value is a nifty useful dial that could assist you with optimizing your Private Cloud’s memory compute resource. It’s the best to understand the minimum RAM that needs to be reported by the OS as available Installed RAM and set the Startup Value respectfully and let Hyper-V Dynamic Memory does its magic and be efficient in RAM distribution amongst your VMs. So long…

More Info:

Scripting dynamic memory, part 5: changing minimum memory

Dynamic Memory Coming to Hyper-V Part 6…

There are lots of excitements at MMS 2013 (#MMS2013) in Las Vegas. Many opportunities to learn about Microsoft Private and Public Clouds, Cloud OS and celebrate many milestones achieved by Windows Server 2012, System Center 2012 SP1, Windows Intune, Windows Azure and Microsoft Cloud Solutions in general.

If you didn’t get a chance to attend, the good news is that you can stream the keynotes live on MSDN Channel 9;

https://channel9.msdn.com/Events/MMS/2013

Also, don’t miss the opportunity to go back and watch the recording of session you like to know more about during the week.

Free Books

Free Windows Server 2012 (RTM) eBook is Available Now

Mitch Tulloch has updated his very popular free ebook on Windows Server 2012 based on the RTM version of the software. A key feature of this book is the inclusion of sidebars written by members of the Windows Server team, Microsoft Support engineers, Microsoft Consulting Services staff, and others who work at Microsoft. These sidebars provide an insider’s perspective that includes both “under-the-hood” information concerning how features work, and strategies, tips, and best practices from experts who have been working with the platform during product development. If you are into eReaders, you can also download the ePub and Mobi versions for free.

Free ebook: Introducing Windows 8: An Overview for IT Professionals (Final Edition)

Get a headstart evaluating Window 8 - guided by a Windows expert who’s worked extensively with the software since the preview releases. Based on final, release-to-manufacturing (RTM) software, this book introduces new features and capabilities, with scenario-based insights demonstrating how to plan for, implement, and maintain Windows 8 in an enterprise environment. Get the high-level information you need to begin preparing your deployment now.

• Topics include:
• Performance, reliability, and security features
• Deployment options
• Windows Assessment and Deployment Kit
• Windows PowerShell™ 3.0 and Group Policy
• Managing and sideloading apps
• Internet Explorer® 10
• Virtualization, Client Hyper-V, and Microsoft Desktop Optimization Pack
• Recovery features

More Resources

Real impact for lean government

https://www.microsoft.com/industry/government/guides/Lean-Gov/default.aspx

Microsoft Private Cloud Solutions

https://www.cloudassessmenttool.com

Did you Know these facts?

https://www.didyouknow2012.com

Making government lean with virtualization and private cloud

https://www.microsoft.com/government/ww/public-services/blog/Pages/post.aspx?postID=255&aID=66

Microsoft Government

https://www.microsoft.com/government/en-us/state/pages/default.aspx

Microsoft Government Cloud Computing

https://www.microsoft.com/industry/government/guides/cloud_computing/default.aspx

Microsoft Government Server Virtualization

https://www.microsoft.com/industry/government/solutions/Server_Virtualization/default.aspx

Free Books

Free Windows Server 2012 eBook Available Now

Download your free copy of the Microsoft Press eBook Introducing Microsoft Windows Server 2012. Written by Mitch Tulloch with the Microsoft Windows Server Team, this book explores enhancements and new features in Windows Server 2012 RC. If you are into eReaders, you can also download the ePub and Mobi versions for free.

Free SQL 2012 eBook Available Now

Download your free copy of the Microsoft Press eBook Introducing Microsoft SQL Server 2012. Written by Stacia Misner and Ross Mistry, this book explores enhancements and new capabilities, including improvements in operation, reporting, and management.

More Resources

https://www.microsoft.com/government/en-us/state/pages/default.aspx

https://www.microsoft.com/industry/government/guides/cloud_computing/default.aspx

https://www.microsoft.com/industry/government/solutions/Server_Virtualization/default.aspx

Hyper-V differencing virtual hard disk (VHD) is an interesting and sometimes very useful subject in virtualization world.

As a matter of fact Snapshot AVHDs are also differencing virtual hard disks. My favorite scenarios for using snapshots are stateless VDI (Virtual Desktop Infrastructure), training VMs and short term test labs. Differencing VHDs provide us with deployment speed and great storage saving.

If you need to know more about creating Differencing disks, please see the following article;

Hyper-V Virtual Machine (VM) Parent-Child Configuration Using Differencing Disks

https://social.technet.microsoft.com/wiki/contents/articles/hyper-v-virtual-machine-vm-parent-child-configuration-using-differencing-disks.aspx

One of my outstanding questions regarding Snapshots and Diff Disks has been walking the chain of child and parents up to the top parent. Off course, you can use Hyper-V Management Console and use Inspect Disk option to find the parent but you might be looking for another method to programmatically walk the chain and show you the relationship of multiple differencing disks in a tree. This made me look at few WMI classes and write few lines of code.

The first step is to find the AVHD (Snapshot) or VHD (Diff Disk) that is the newest one in the directory. A simple dir (PowerShell alias for Get-ChildItem) and Sort-Object will take care of this.

The second step is to inspect the VHD and you can take care of it with GetVirtualHardDiskInfo method of Msvm_ImageManagementService WMI class under Root\Virtualization namespace

Get-WmiObject –ComputerName “Localhost” -Namespace root\virtualization -Class Msvm_ImageManagementService

The output of this method, shows the parent of virtual hard disk being inspected in XML format. You can retrieve the parent virtual disk name and use it as the input for the next round of check. like a recursive loop.

So, I created the following function to take care of this repetitive task;

Function Inspect-DiffVHD([string]$VHDPath,[string]$Server = ".")  

{

    $HyperVNamespace = "root\virtualization"

    $Server = "."

    $ImageManagementServiceName = "Msvm_ImageManagementService"

    $ImageManagementService = Get-WmiObject -ComputerName $Server `

       -Namespace $HyperVNamespace -Class $ImageManagementServiceName

    $result = $ImageManagementService.GetVirtualHardDiskInfo($vhdPath)

    $VHDInfo = [xml]$result.info

    $VHDInfo.INSTANCE.PROPERTY[4].value

}

The attached sample PowerShell script would generate the following output when pointed to a folder containing differencing VHDs (.VHD) or Snapshots (.AVHD). (The sample script assumes the original Child/Parent relationship hasn’t been altered).

Sample output;

Enter Path: D:\Hyper-V\Test

D:\hyper-v\test\test_72A34A5A-0B86-4858-9663-CCE3AD9828A2.avhd

  D:\Hyper-V\test\test_9330B5ED-248D-442D-9705-F7A320F7AC62.avhd

    D:\Hyper-V\test\test_F6BC8CA6-BC70-4D02-9D6D-5E3C7CAB9D6E.avhd

      D:\Hyper-V\test\test_84D8C8BC-1723-49DB-92D8-CEC1752ADC3F.avhd

        D:\Hyper-V\test\test.vhd

Please note that I didn’t get a chance to test my sample script against multiple tree snapshot scenario but you should be able to do this with a simple modification.

Hope you find this post helpful. So long…

This question comes up frequently. So, how do we size the pagefile on Hyper-V host servers and how about Virtual Machines?

Here’s the all you should know;

Paging File Configuration within the VM

Proper paging file configuration is vital to the way DM performs

DM requires that the VM have a PF

So how do you size your Paging File?

The old way doesn’t apply to DM:

• Use Peak Commit Charge
• Size PF to Peak Commit Charge – Physical Memory + some buffer

To understand how we should size the PageFile (PF), let’s consider an example:

Current Commit: 100 MB

Free Buffer: 20%  (this is a new Dynamic Memory Setting)

Target Commit: 125 MB

Paging File: 1MB

Commit Limit: 126 MB

Available Memory: 26 MB (Commit Limit – Current Commit)

What happens when an app allocates 50 MB of memory?

What happens when an app allocates five 10 MB memory chunks?

Guidance:

Minimum PF should be large enough to cover the memory demands of your largest process. In the case of application allocating 50 MB of memory, the allocations request will fail with out of virtual memory error. When application allocates 10 MB chunks, the first two allocations will go through while the rest will fail as there’s no virtual memory (either RAM or PF) to accommodate that.

Maximum PF should be (peak commit charge – maximum physical memory + some buffer)

Paging File Configuration on the host

If you have reserved enough RAM for the host operations and your host is only being used for Hyper-V (plus Fail Over Cluster for HA VMs), you shouldn’t need a large pagefile.

With Dynamic Memory, you have to make sure that your host reserve is set properly

HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Virtualization

                Value Name: MemoryReserve

                Value Type: DWORD

                Value Data: (Decimal)Memory to reserve in MB

With a properly configured host reserve, if you follow (Peak Commit Charge – Physical Memory + some buffer) formula on the host, you’ll find out that you don’t need a page file.
So, Pagefile size of min= few Gigabyte and Max=2xmin should be sufficient.

Also, there are 2 new performance counter objects available to the Hyper-V host;

Hyper-V Dynamic Memory Balancer

• Provides counters about the memory balancer running in the parent partition
• Counters are focused on:
• The memory that is available for the host to use
• The add and remove operations performed by the memory balancer

Hyper-V Dynamic Memory VM

• Provides counters about memory usage on the guest VMs that are currently running
• Each VM has its own set of counters
• Counters are focused on:
• Memory added and removed from the guest
• The memory pressure that the guest is experiencing

Hope this helps! so long…

In this post, we're going to look at an issue that was happened when I was teaching a Hyper-V R2 class few weeks back.

Students were experimenting Hyper-V Role installation options using Server Manager Console, DISM.exe, ocsetup.exe and ServerManager PowerShell Module.

After reboot, we noticed that one of Windows Server 2008 R2 SP1 Full edition hosts (Not Core) didn't have Hyper-V Management Console listed under Administration Tools. In addition to that, there no Hyper-V Role listed under Server Manager Console Roles!

When we tried to select the Hyper-V Role in Server Manager Add Roles Wizard, the following error message surprised us!

Hyper-V cannot be installed

The processor on this computer is not compatible with Hyper-V. To install this role, the processor must have a supported version of hardware-assisted virtualization, and that feature must be turned on in the BIOS.

We verified that hardware-assisted virtualization feature and hardware DEP are enabled within the server's BOIS.

Bcdedit.exe showed Hypervisorlaunchtype was set to Auto

%windir%\logs\CBS.log showed Hyper-V roles is installed and didn't show any errors.

Looking at dism.exe feature query (dism.exe /online /get-features) showed

Feature Name : Microsoft-Hyper-V

State : Enabled

So, comparing with a working Hyper-V R2 (Full Edition) host we found that the following feature was not installed in here;

Microsoft-Hyper-V-Management-Clients

So, it turned out the he’d used the following command to install Hyper-V role without installing the management console feature.

This feature is normally being installed on Hyper-V Server Full Edition when using Server Manager Role installation wizard while it needs to be installed separatly if command line tools such as dism.exe, ocsetup.exe are being used.

The command that was used was

Dism.exe /online /enable-feature /featurename:Microsoft-Hyper-V

Note: this command works fine on Server Core edition of R2 as it doesn't require management console and there's no server manager console either.

Instead, the following command should've been used on Windows Server 2008 R2 Full Edition;

Dism.exe /online /enable-feature /featurename:Microsoft-Hyper-V /featurename:Microsoft-Hyper-V-Management-Clients

Hope this helps you prevent this mystery or solve it! So long...

Yesterday while working on installing a new SCVMM 2008 R2 server setup, we encountered an issue when tried to add a new 2-node Hyper-V R2 host cluster to the VMM console.

Here’s how we were able to reproduce the error;

In SCVMM 2008 R2 console, select  Add a host that is “Windows Server-based host on an Active Directory domain” . Enter the name of the server and choose add. You will get the following error;

Hostname.Hosts.domainname.com cannot resolve with DNS.

Ensure there is network communication with the DNS server. If the problem persists, contact your network administrator.

ID: 404
Details: No such host is known

We researched the issue and saw the following Wiki page that did not resolve this particular problem. (This article addresses similar error message in disjointed namespaces)

A few notes on System Center Virtual Machine Manager 2008 and disjointed namespaces

https://social.technet.microsoft.com/wiki/contents/articles/a-few-notes-on-system-center-virtual-machine-manager-2008-and-disjointed-namespaces.aspx

Here’s the TechNet explanation of this error message;

SCVMM 2008 R2 Command-Line Interface Error Messages

https://technet.microsoft.com/en-us/library/dd548296.aspx

404

%ComputerName; cannot resolve with DNS.

Ensure there is network communication with the DNS server. If the problem persists, contact your network administrator.

NSLookup query against all node names and Failover Cluster name proved our DNS was working fine.

The next step was to check the networking and we found that default gateway on the SCVMM Server wasn’t configured properly. Setting the default gateway to the correct gateway resolved this issue.

I’ve also seen other networking problems on SCVMM server could result in the the same error message.

So, here are few troubleshooting steps that might help resolve this;

1- Domain functional level must be at least Windows 2003

2- DNS entries should exist for the SCVMM server and systems that will be added as Nodes and Failover Cluster names if any.

3- Check the network configuration on the SCVMM server such as default gateway and DNS server IP Addresses. Also, if there are multiple NICs on the server, try adding the node while other non-production NICs are disabled.

4- While adding the node to SCVMM, check the “Skip Active Directory name verification” checkbox on the Select Host Servers page where the host name is entered.

So long….