recently, I had some occurrences of the following error while scanning the compliance state of particular hosts or clusters from SCVMM:

Error (2927)

A Hardware Management error has occurred trying to contact server <hostname>  .

WinRM: URL: [http://<hostname>:5985], Verb: [INVOKE], Method: [ScanForUpdates], Resource: [http://schemas.microsoft.com/wbem/wsman/1/wmi/root/scvmm/UpdateManagement]

Unknown error (0x80338043)

Recommended Action

Check that
WinRM is installed and running on server <hostname>. For more Information use the command "winrm helpmsg hresult" and http://support.microsoft.com/kb/2742275

This happens because the answer of the host with the installed Hotfixes grows larger and needs to be fragmented. Here WinRM Needs to be updatedt to a later version.

To solve this problem you need to install “Windows Management Framework (WMF) 5.0” at least on your SCVMM Server.

At the time of this writing the Production Preview of WMF 5.0 is available at https://www.microsoft.com/en-us/download/details.aspx?id=48729

Cheers

Robert

now that UR8 is available, I modified my script slightly to also check if the DHCP Extension is correct. If it is shown in red, please manually update it on every host.

Usually, I expected that my hosts would report that they need attention and a Host Agent update would be required. As this did not happen, I wrote below script to verify the latest Agents are installed, and optionally offer to do the update.

Please note that before doing the update, you should take the usual precautions to prepare for Host and VM Downtime. Also, you need to have Execution Policy set to allow running PS scripts.

Copy and paste below script into checkagentversion.ps1

#-----------------------------------
# Check Version of VMM Host agents v2.
#-----------------------------------
Write-Host "RTM 3.2.7510.0"
Write-Host "UR1 3.2.7620.0"
Write-Host "UR2 3.2.7634.0"
Write-Host "UR3 3.2.7672.0"
Write-Host "UR4 3.2.7768.0"
Write-Host "UR5 3.2.7995.0"
Write-Host "UR6 3.2.8002.0"
Write-Host "UR7 3.2.8071.0"
Write-Host "UR8 3.2.8117.0"

$vmmserver = get-scvmmserver -ComputerName "localhost"
Write-Host "This SCVMM Server Version"  $vmmserver.ProductVersion

$version = $vmmserver.ProductVersion
$outdated = 0

$ManagedHosts = Get-SCVMMManagedComputer

foreach ($Managedhost in $managedHosts)
{

    if ($version -eq $Managedhost.AgentVersion)
    {
        write-Host $Managedhost.ComputerName $Managedhost.AgentVersion
    }
    else
    {
        write-Host $Managedhost.ComputerName $Managedhost.AgentVersion -ForegroundColor Red
        $outdated = 1
    }

    $DHCPExtn = Get-WmiObject -Class win32_product -Filter 'Name = "Microsoft System Center Virtual Machine Manager DHCP Server (x64)"' -computername $Managedhost.ComputerName
   
    if ($DHCPExtn)
    {
            $seperation = ($DHCPExtn.Version).Split("{.}")
            $dhcpversion = [int]$seperation[2]
            if ($dhcpversion -ge 8071)
            {
                write-Host " DHCPExtn " $DHCPExtn.Version
            }
            else
            {
            write-Host " DHCPExtn " $DHCPExtn.Version "Manual uninstall/install required" -ForegroundColor Red
            }
     }
    else
    {
        write-Host " No DHCPExtn "
    }
   

}

if ($outdated -eq 1)
{
    write-host
    Write-Host "Some Managed Hosts have not the latest agent installed"-ForegroundColor Red
    Write-Host "Warning: Updateing now may cause downtime to the Host and the running VMs"-ForegroundColor Red
    $message= "Do you want to update hosts now.?"
    $yes = New-Object System.Management.Automation.Host.ChoiceDescription "&Yes",""
    $no = New-Object System.Management.Automation.Host.ChoiceDescription "&No",""
    $choices = [System.Management.Automation.Host.ChoiceDescription[]]($yes,$no)
    $result = $Host.UI.PromptForChoice("Update All Hosts with latest Agent",$message,$choices,0)

    # Do the update

    if($result -eq 0)
    {
   
       $Credential = Get-Credential
       foreach ($Managedhost in $managedHosts)
       {
            if ($version -ne $Managedhost.AgentVersion)
            {
            Write-Host "Updating "  $Managedhost.ComputerName
            $VMMManagedHost = Get-SCVMMManagedComputer -ComputerName $Managedhost.ComputerName
            $result=Update-SCVMMManagedComputer -VMMManagedComputer $VMMManagedHost -Credential $Credential
           
            }
       }
   
    }

}
else
{
    write-Host "Agents on managed computers are up to date"
}

# End of script

Additionally, if you are using Network Virtualization check out this blog to ensure having the latest DHCP extenions. http://blogs.technet.com/b/scvmm/archive/2014/07/31/support-tip-vms-deployed-to-hyper-v-networks-experience-delays-acquiring-an-ip-address-after-reboot.aspx

Hope this helps

Cheers

Robert

recently I had to build an environment with Hyper-V virtual fibre channel. As I’m not a SAN expert I had some key learnings that I’d like to share with you.

A good lecture is http://social.technet.microsoft.com/wiki/contents/articles/18698.hyper-v-virtual-fibre-channel-troubleshooting-guide.aspx

So I used a 2 node cluster with one Emulex LP adapter in each node. HA enviroments usually have at least 2 or more.

Firs,t use Hyper-V Manager, Virtual SAN Manager, to Create a New virtual Fibre Chanel Switch

Do that on all cluster nodes with identical Name. Don’t forget to make the check box to link this virtual switch to your HBA.

Second, configure the physical HBA to use NPIV. The Tool for Emulex is One Command Manger (OCManager)

Do this on all  nodes. Also ensure to have current Firmware up to date. For Troubleshooting check if Adapter is capable to be used (gwmi -n root\virtualization\v2 Msvm_ExternalFcPort -Property InstanceId, WWPN, IsHyperVCapable)

Third, add a Fibre Channel Adapter to the VM, and configure Word Wide Port Names.

Of importance is the World Wide Port Name (WWPN). This is the End to End connection for the SAN LUNs and must be made known to your switches (zones) and in your Storage to map the LUN.

We use 2 WWPNs to allow for faster migration, so the second WWPN can be made active on the destination while the migration is ongoing. This WWPN is reported to the HBA so that he actually represents the virtual port in addition to his physical port. Failure to open this port on the HBA result in a failure to  tart the VM like this:

'vRDGW' Synthetic FibreChannel Port: Failed to start reserving resources with Error 'Insufficient system resources exist to complete the requested service.'(0x800705AA).

'vRDGW': Operation for virtual port (C003FFEFACDC0006) failed with an error: No physical port available to satisfy the request (Virtual machine ID D7AAA601-D3C7-4007-AC9D-C0AA1B087CC5).

This failure was resolved for me with actual FW on the HBA. You also see this failure when the HBA is not connected to a NPIV Switch.

Hyper-V suggests a WWPN out of a pool that you can modify with Virtual SAN Manager. This has to be unique in your SAN

Hyper-V also suggests a WWNN which is always C003FF0000FFFF00. This does not seem to be an issue, however you may see complaints from Cluster Verifier. So use different WWNN for each VM.

During startup of the VM LUNs are detected. If no LUN is available this might be because the WWPN is not correctly in your Zoning, or not mapped on the Storage. This causes the startup to hang 90 seconds at 1%

'vRDGW': No LUNs have appeared for Synthetic Fibre Channel HBA Fibre Channel Adapter (F5F8DD0B-5EF9-4958-9AF3-48FBF3A6C505). The VM has been started after the timeout period (90 seconds).

 Now, you should see the LUNs appear in your VM, that needs to be a Server OS. You may see the LUN multiple times due to multiple pathes in your SAN. So usually you will need to install the MPIO and DSM of our storage vendor.

I hope that may help you to get started with vFC.

Cheers

Robert

I'd like to point you to a Whitepaper from your Development Group that impressively shows how to build High Performance Hyper-V Clusters today.

http://www.microsoft.com/en-ie/download/details.aspx?id=42960

Cheers

Robert

A common question today is how to setup your cluster blades with (10GB) Ethernet connections. How do I configure that correctly with the following requirements:

• A LBFO (Load Balanced FailOver) Team of the available physical Connections.
• Several Networks which are controlled by QoS to not use up the entire channel.

Management Network

Cluster Network

iSCSI Network

Live Migration Network

(Optional) multiple isolated virtual networks

Let’s see how this can be build using SCVMM 2012 R2.

I start with 2 Nodes. Each has 3 Ethernet interfaces. Ethernet1 is already used by a virtual switch (NodeSwitch). The Switch used Ethernet1 as a team with one NIC.
A virtual NIC vEthernet(Management) is exposed to give the Host Network connectivity

 
This Setup makes it easier to build the desired switch with the 2 remaining NICs, as I will always have connectivity during the creation of the new switch. Later, I can easily destroy the NodeSwitch and add Ethernet1 to the newly build Switch as 3^rd NIC.

Let’s start building:

1. Define the Logical Networks the new Switch should connect to.

The Wizard allows you to already create an identical VM Network right here. As you always need a VM Network mapped to one logical Network let’s use this.

First creating the “Management” Logical Network:

Add a Network site for this Logical Network. Here is the IP Address range for my management purposes. It matches my DHCP Scope, so I do not specify that again here.

Do the same steps to create the other Networks. To allow separation and QoS of those connections you have to work with own subnets on top of the existing network.

I use:

Tenant (172.31.10/24), Cluster (172.31.2.0/24),  iSCSI (192.168.1.0/24) , Live Migration (172.31.3.0/24)

When adding the Network Sites for those specify the IP Subnet.

For the Tenant Network, specify to use Network virtualization and don’t create the VM network yet.

As it uses Network virtualization we can connect several isolated VM Networks later (Red and Blue)

That results in the following logical Networks

And corresponding VM Networks:

Back on logical Networks, create IP Pools on each Network so that VMM can give that out statically later. (I just accepted the defaults, after giving it a corresponding name…)

Just as Management Network is using DHCP in my case, no need for an IP Pool.

Now as we have created these Networks, we can start building the Switch Template using those.

(You can later create new Networks, and associate it with the Template, if you have to.)

2. Define the Uplink Port Profile for your new switch.

We need to specify which networks are reachable by the switch.

As I used the switch for VM’s mainly, I use Hyper-V Port balancing.

Next, add all Network Sites this switch will be connected to, and enable Network virtualization too

 NodeLN_0 belongs to my other Switch. So not connecting it here.

3. Create new Virtual Switch Template (Name ClusterNodeSwitch)

Here the important part is to select Uplink mode, and Add the previously created Uplink Port Profile

Next, we need to specify which Virtual Ports with which qualification should be available when later placing the switch. You could basically include every type here, and pic later the ones you need.

I did this for the possible networks I created previously. The Default helps to have the field prepopulated later when I assign VM Networks to VMs.

4. Finally, placing the switch on the nodes.

 Here’s one of my hosts properties before placing the switch: (as this was pre-existing, to how the differences later. There is no need to have this)

Start adding a New Logical Switch

Select the right Switch Template, and adding the physical NICs

Now, create the Virtual Network Adapters the Host should have available for it's use. Otherwise it would have no connectivity.

I chose not to inherit the settings, as I have a already existing management connection. This is required if you have no other connection yet.

For all Networks I selected to pick a Static IP from the suggested pool

Do the same on the other nodes….

So in NCPA on one node this now shows as:

Ipconfig:

All those Networks also show up in Failover Cluster Manager. (I’ve given them the correct names based on the used IP already.

Now, select the correct Network for LiveMigration:

5. (Optional): Create Isolated BLUE and RED VM Networks

To run virtual networks with identical subnets, isolated, you create VM Networks that map to the Tenant Logical Network.

Give it a subnet that might seem to collide with your on premises or other VM Networks, but it’s isolated.

And we create an associated IP Pool, so that we can assign later IP addresses to the VM and they will receive them from a DHCP extension installed in every Hyper-V switch.

Doing the same for Blue, creates us two isolated VM networks.

Now, to see that in action, I assign  VMs to 3 different networks:

First, to my Management Network, it will get a DHCP address from there

Second, to the Blue Network, DHCP on the Hyper-V Switch will give out IP address here

Lastly, to the Red Network, DHCP on the Hyper-V Switch will give out IP address here of the Red scope:

I hope that helps you better understand and use this cool feature. It turns out handy when you configure more then just 2 nodes.

Cheers

Robert

recently, I had an issue where it was needed to carefully control which machines can talk to each other by using a feature in Server 2012: Port Access Contol Lists (ACLs)

The feature is described in http://technet.microsoft.com/en-us/library/jj679878.aspx but might require more explanation. You can use the ACLs on MacAddresses or IPAddresses and additionaly specify Local or Remote. This should be considered Local or Remote from the VMs perspective. So the example in the above link actually disables all MacAddresses but only allows one local Mac inside the VM (and it is missing one Byte)

Let me give you another example, that limits the IP traffic to only one other machines.

W12testvm1 has IP 10.0.0.131 and it should only communicate with W12testvm2 which has IP 10.0.0.160
The PS commands are issued on the host running w12testvm1

1. „Block all IPs outgoing from w12testvm1“
add-VMNetworkAdapterAcl  -VMName w12testvm1 -LocalIPAddress any -Direction Both -Action deny

2. „Allow the IP of w12testvm1 to be sent and received“
  add-VMNetworkAdapterAcl  -VMName w12testvm1 -LocalIPAddress 10.0.0.131 -Direction Both -Action allow

3. „Block all IPs incoming to w12testvm1 “
add-VMNetworkAdapterAcl  -VMName w12testvm1 -RemoteIPAddress any -Direction Both -Action deny

4. „Allow only the IP of w12testvm2 incomming“
add-VMNetworkAdapterAcl  -VMName w12testvm1 -RemoteIPAddress 10.0.0.160 -Direction Both -Action allow

 To now query the current setting use

PS C:\Windows\system32> Get-VMNetworkAdapterAcl -VMName w12testvm1

VMName: w12testvm1

Direction    Address                                                  Action           
---------        -------                                                      ------           
Inbound      Local  10.0.0.131                                   Allow            
Inbound      Local  0.0.0.0/0                                      Deny             
Inbound      Local  ::/0                                               Deny             
Inbound      Remote ::/0                                            Deny             
Inbound      Remote 10.0.0.160                                Allow            
Inbound      Remote 0.0.0.0/0                                   Deny             
Outbound     Local  ::/0                                             Deny             
Outbound     Local  10.0.0.131                                 Allow            
Outbound     Local  0.0.0.0/0                                    Deny             
Outbound     Remote ::/0                                          Deny             
Outbound     Remote 0.0.0.0/0                                 Deny             
Outbound     Remote 10.0.0.160                              Allow

Hope you find this usefull

Cheers
Robert

I just had a interesting task I'd like to share:

Customer had some Linux CentOS 6.3 VMs running on VmWare ESX. The task was to convert this machines to Windows Azure VMs. Here are the steps we made:

1. For the destination Platform Azure customer installed the Linux Integration Services into CentOS before starting the conversion. http://www.windowsazure.com/en-us/manage/linux/common-tasks/upload-a-vhd/  Step 4: Prepare the image to be uploaded

 
2. We then copied all files belonging to the VM(vmdk, vmx,nvram,….) to the Library Share of SCVMM 2012 SP1 installation

 
 Two VMs in the SCVMM Library

3. From SCVMM we started to „Create a Vitual Machine“, this time „Convert Virtual Machine“ and selected the VM from the Library. This converts the VMware installation to a Hyper-V VM (with vhd) and deploys it to a Hyper-V host.

SCVMM does not know how to modify Linux VMs,so this Warnings are expected.

The VM with the Linux Integration Services, starting on Hyper-V. The WALinuxAgent tries to detect Azure Environment.

4. As Azure requires fixed size VHD format, we used Hyper-V Manager „Edit Disk“, selected the previously converted vhd, and converted this to a new fixed size vhd.

5. With App Controller 2012 SP1, we added the Share which holds the vhd. This allows to simply copy and paste the VM from the local share to the Azure Storage Blob.

6. Within Azure Portal, we created a Disk from this VHD, and build a VM with this Disk. (That could also be done with App Controller if you like)

Cheers

Robert

I just had an issue that my 2012 Hyper-V Hosts did not have the new powershell commands available. (get-vm, etc)

I installed the Hosts using SCVMM 2012 SP1 (beta) using Bare Metal Deployment http://blogs.technet.com/b/cloudytom/archive/2012/01/02/scvmm-2012-bare-metal-deployment-checklist.aspx

As it turned out, this does not add the optional hyper-v powershell module.

To add them afterwards, use the following powershell command:

install-windowsfeature -name hyper-v -IncludeManagementTools

Cheers

Robert

recently, I had an issue with a RHEL 6.3 VM with the latest Integration Components 3.4. After live migration of the VM to another node, it was not reachable over the network anymore.

Usually, the ICs send out a  "Gratuitous ARP" to let each switch update their internal routing table  to the VM. The ICs depend on a notification of the Linux kernel, which has to be enabled in RHEL 6.3

A solution to this problem is to edit  the file, “/etc/sysctl.conf” inside the RHEL VM, and insert the following line, then reboot.

net.ipv4.conf.default.arp_notify = 1

Cheers

Robert