• Operating systems are already installed on at least four machines (SqlNode1, SqlNode2, VmmNode1, VmmNode2)
• Failover Clustering is configured so that SqlNode1 and SqlNode2 are in one cluster and VmmNode1 and VmmNode2 are in a different cluster.
• SQL Enterprise is installed and configured with always on capability enabled and a listener configured (SqlFMList). In our case, the listener was configured to listen on port 1771. SQL must be using port 1433 for a non-always on instance / database for this to work

The installation steps we used

1. Create the VMMService account (eg svc_vmmservice), and grant it permissions to
   • SQL on SqlNode1
   • Local administrator group on  VmmNode1 and VmmNode2
2. Install VMM on VmmNode1. When you get to the SQL configuration section, give the computer name SqlNode1, port 1433, and allow it to create the database (VirtualManagerDB). Note down the location of the DKM.
3. Uninstall VMM, choosing the option to retain the database.
4. Ensure the svc_vmmservice account has a db_owner permissions of the VMM database.
5. Ensure the SQL Listener / AG is active on the node that has the Database, but do not add the database to the Always On group yet.
6. Use the below ini file to install VMM on one VMM node at a time -  (we couldn't get this to work using the graphical installer due to the SQL port required and issues with the user interface). This will install VMM so that when launching the console, connect to the address "vmm:8100"

[OPTIONS]
ProductKey=XXXXXXXXXXXX
UserName=Name
CompanyName=Name
ProgramFiles=C:\Program Files\Microsoft System Center 2016\Virtual Machine Manager
CreateNewSqlDatabase=0
SqlDatabaseName=VirtualManagerDB
SqlServerPort=1771
RemoteDatabaseImpersonation=0
SqlMachineName=SqlFMList
IndigoTcpPort=8100
IndigoHTTPSPort=8101
IndigoNETTCPPort=8102
IndigoHTTPPort=8103
WSManTcpPort=5985
BitsTcpPort=443
#CreateNewLibraryShare=1
#LibraryShareName=MSSCVMMLibrary
#LibrarySharePath=D:\VMMLIB\
#LibraryShareDescription=Virtual Machine Manager Library Files
SQMOptIn = 0
MUOptIn = 0
VmmServiceLocalAccount = 0
TopContainerName="CN=DKM,DC=LAB,DC=INTRANET" #replace this with the DKM location
HighlyAvailable = 1
VmmServerName=VMM
VMMStaticIPAddress=10.51.83.236
Upgrade=0

The command line used to use this INI file is:

setup.exe /server /i /f <IniFilePath> /VmmServiceDomain <Domain> /VmmServiceUserName <Service Account> /VmmServiceUserPassword <Service account PW> /IACCEPTSCEULA

7. After installation of VMM on both nodes, check the VMM console can connect to the server
8. Using cluster manager take the VMM service offline
9. Add the SQL database to the AG with the listener used in the above file (SqlFMList).
10. Restart VMM. Check that VMM works with different combinations of which node VMM & SQL are on.

• From VMM server:

Cd "C:\Program Files\Microsoft System Center 2016\Virtual Machine Manager\agents\Linux"
c:\program files\putty\PSCP.exe *.* username@VmIp:

• the ':' after the IP is important - specifies the location to copy the files to. without anything after the ':' it will copy it to the users' home folder
• Then from VM:

# Update with latest security & other updates
sudo apt-get update && sudo apt upgrade
# enable automatic upgrades
sudo apt install unattended-upgrades
cat > 20auto-upgrades << EOF
APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "1";
APT::Periodic::AutocleanInterval "7";
APT::Periodic::Unattended-Upgrade "1";
EOF
sudo mv ./20auto-upgrades /etc/apt/apt.conf.d/
# install VMM agent
chmod +x install
sudo ./install ./scvmmguestagent.1.0.2.1075.x64.tar
# UEFI fix (needed for Gen 2 VMs)
sudo grub-install --target=x86_64-efi --efi-directory=/boot/efi --no-nvram --removable
# install & enabled Hyper-V Integration components
sudo apt-get update && sudo apt-get install linux-tools-$(uname -r) linux-cloud-tools-$(uname -r) -y
hv_fcopy_daemon
hv_kvp_daemon
hv_set_ifconfig
hv_vss_daemon
hv_get_dns_info
# shutdown VM for VHD image. (this is also needed for IC's to start working)
sudo shutdown -h now

• Next copy VHD to VMM library
• Set VHD OS type to Ubuntu 16.04 (64 bit)
• Create VM template
  • Make sure secure boot is disabled & VM Generation matches that used above.

#get Secret object (Containing private key) from Key Vault
$AzureKeyVaultSecret=Get-AzureKeyVaultSecret -VaultName $VaultName -Name $CertificateName -ErrorAction SilentlyContinue

#Convert private cert to bytes
$PrivateCertKVBytes = [System.Convert]::FromBase64String($AzureKeyVaultSecret.SecretValueText)

#Convert Bytes to Certificate (flagged as exportable & retaining private key)
#possible flags: https://msdn.microsoft.com/en-us/library/system.security.cryptography.x509certificates.x509keystorageflags(v=vs.110).aspx
$certObject = New-Object System.Security.Cryptography.X509Certificates.X509Certificate2 -argumentlist $PrivateCertKVBytes,$null, "Exportable, PersistKeySet"


#Optional: import certificate to current user Certificate store
$Certificatestore = New-Object System.Security.Cryptography.X509Certificates.X509Store -argumentlist "My","Currentuser"
$Certificatestore.open("readWrite")
$Certificatestore.Add($certObject)
$Certificatestore.Close()

#if private certificate needs to be exported, then it needs a password - create Temporary Random Password for certificate
$PasswordLength=20
$ascii = 33..126 | %{[char][byte]$_}
$CertificatePfxPassword = $(0..$passwordLength | %{$ascii | get-random}) -join ""

#Encrypt private Certificate using password (required if exporting to file or memory for use in ARM template)
$protectedCertificateBytes = $certObject.Export([System.Security.Cryptography.X509Certificates.X509ContentType]::Pkcs12,
$CertificatePfxPassword)
Write-output "Private Certificate Password: '$CertificatePfxPassword'"

#Optional: Export encrypted certificate to Base 64 String in memory (for use in ARM templates / other):
$InternetPfxCertdata = [System.Convert]::ToBase64String($protectedCertificateBytes)

#Optional: Export encrypted certificate to file on desktop:
$pfxPath = '{0}\{1}.pfx' -f [Environment]::GetFolderPath("Desktop") ,$CertificateName
[System.IO.File]::WriteAllBytes($pfxPath, $protectedCertificateBytes)

function Get-AuthHeaderUsingCert {
    param(
        $TenantID,
        $CertificateThumbprint,
        $ApplicationIdURL,
        $ApplicationId
    )
    $AuthUri = "https://login.windows.net/$TenantID/oauth2/authorize"
    $clientCertificate = Get-Item -Path Cert:\CurrentUser\My\$CertificateThumbprint
    $authenticationContext = New-Object -TypeName Microsoft.IdentityModel.Clients.ActiveDirectory.AuthenticationContext -ArgumentList $AuthUri
    $certificateCredential = New-Object -TypeName Microsoft.IdentityModel.Clients.ActiveDirectory.ClientAssertionCertificate -ArgumentList ($ApplicationId, $clientCertificate)
    $authToken = $authenticationContext.AcquireTokenAsync($ApplicationIdURL, $certificateCredential)
    $authHeader = @{
        'Content-Type'='application/json'
        'Authorization'=$authToken.result.CreateAuthorizationHeader()
    }
    $authHeader
}

add-type -path C:\Temp\Microsoft.IdentityModel.Clients.ActiveDirectory.dll
#I'm using Version 2.28.31117.1411 of this DLL - Apparently newer ones don't work for this.

$AuthHeader = Get-AuthHeaderUsingCert -TenantID $tenantID -CertificateThumbprint $CertificateThumbprint -ApplicationId $ApplicationId -ApplicationIdURL $ApplicationIdURL 
if([string]::IsNullOrWhiteSpace($authheader)){throw "no auth"}
$JSONArguments = $(@{'ArgName'='ArgValue'} | convertto-json -Depth 5)
$Result = Invoke-Webrequest -Uri $WebhookURL -Headers $AuthHeader -Body $JSONArguments -Method Post
$result

param([switch]$runSysprep=$false)
write-output "Sysprep Script Run, parameter 'runSysprep': $runSysprep"

if($runSysprep){
  write-output "starting Sysprep"
  Start-Process -FilePath C:\Windows\System32\Sysprep\Sysprep.exe -ArgumentList '/generalize /oobe /shutdown /quiet'
  write-output "started Sysprep"
}else{
  write-output "skipping Sysprep"
}

$VMName =  'vmName'
$VMRG = 'vmResourceGroup'
$VMLocation = 'AustraliaEast'
$ExtensionName = 'runsysprep'
$Scripturi = 'https://<storageAccountName>.blob.core.windows.net/templates/scripts/sysprep.ps1'
Set-AzureRmVMCustomScriptExtension -FileUri $ScriptURI -ResourceGroupName  $VMRG -VMName $VMName -Name $ExtensionName -Location $VMLocation -run './sysprep.ps1' -Argument '-runSysprep'

RequestId IsSuccessStatusCode StatusCode ReasonPhrase
--------- ------------------- ---------- ------------
                         True         OK OK

$status = Get-AzureRmVMDiagnosticsExtension -ResourceGroupName $VMRG -VMName $VMName -Name $ExtensionName -Status
$status.SubStatuses.message

Sysprep Script Run, parameter 'runSysprep': True\nstarting Sysprep\nstarted Sysprep

Stop-AzureRmVM -Name $VMName -ResourceGroupName $VMRG  -force

• Ensure that you are applying weights to the network traffic to prioritize traffic types in this order (there are several methods to apply these weights - depending on the network config, and this all changes for Win2016):
  1. Host Management,
  2. CSV storage (Win2012 R2 and earlier)
  3. VM storage
  4. Live Migration (if using dynamic optimization or some other automatic load balancing method - otherwise put this last)
  5. VM traffic
• If the prioritization has already occurred, and any suboptimal performance is occurring, then either:
  • increase capacity (more / faster NICs & switches)
  • Reduce load by shifting workloads to other hosts

Virtual Machine Network throughput

This is a continuation of a series on Hyper-V performance monitoring. The previous posts covered CPU and storage performance, This post is on Memory performance, and the remaining post will cover network performance.

Memory performance in Hyper-V environments is a mix of availability, ensuring Virtual Machines have the memory they need, and NUMA memory usage.

Host Available Memory

• Counter: Hyper-V Dynamic Memory Balancer\Available Memory
• Why: This is the memory available for virtual machine usage on a Hyper-V host, while ensuring that there is sufficient memory available for Hardware and Hyper-V management.
• Threshold: < 2GB
• What to do if threshold is breached:
• Check for many VMs with memory pressure < 60%, if so, reduce min memory
• Check for static memory assignment VMs that have been assigned more than needed (DM candidates?)
• Check Dynamic Memory is responding on virtual machines enabled for Dynamic Memory (a value for the “Average Pressure” value provides that).
• Reduce memory usage on hosts (load balance VMs between hosts?)

Virtual Machine Memory Pressure

• Counter: Hyper-V Dynamic Memory VM(*)\Average Pressure
• Threshold: Any Instance > 85% sustained > 1 min. (Note this thresholds assumes the default buffer setting of 20% - adjust this threshold on a per virtual machine basis if the buffer value is modified)
• Why: This displays the ratio of committed memory within the VM to the memory allocated to the virtual machine. This indicates whether the virtual machine has enough memory for its current needs, and values close to and over 100 indicate performance impacting page file use within the virtual machine. This value will be 0 if dynamic memory is disabled on Hyper-V 2012 R2 and earlier.
  For virtual machines using dynamic memory, Hyper-V should maintain this at "100 / (100 + (100 * Memory Buffer))" (for default buffer for 20%, target pressure is 83.3%)
• What to do if threshold is breached:
• Check each virtual machine's max memory settings (vs how much has been allocated to the virtual machine)
• Check overall host available memory

Virtual Machine Allocated Memory

• Counter: Hyper-V Dynamic Memory VM(*)\Physical Memory
• Threshold:
• no threshold (informational counter, used for capacity planning), or
• (optional) if dynamic memory is enabled, then > 90% of maximum memory. This is optional due the difficulty in implementing this with VMs of different maximum values.
• Why: This displays the current amount of allocated memory for each virtual machine (this value will reduce if dynamic memory reduces the memory allocated to virtual machines). The amount of memory shown within the virtual machine (either task manager or performance monitor counters) does not reduce when the memory is reduced (through dynamic memory), and so should not be used.
• What to do if threshold is breached (if one is used):
• identify if the dynamic memory maximum value for the virtual machine needs to be increased
• check the memory usage by the virtual machine is expected.

Remote NUMA Memory Node Access

• Counter: Hyper-V Vm Vid Partition(*)\Remote Physical Pages.
• Threshold: > 100 (and increasing by at least 100/hour). Note that this value will reset if the VM is restarted or migrated to a different host.
• Why: A Non-Uniform Memory Access (NUMA) node is a physical grouping of processors and memory (on a motherboard) for performance. Accessing memory on a remote NUMA node (on the same motherboard) is slower than accessing memory in the closest NUMA node for any type of CPU processing.  A remote physical page access occurs when a virtual machine CPU core running on one physical processor accesses virtual machine memory on a remote NUMA node. Hyper-V does everything it can do to avoid this occurring by choosing which processors and sections of physical memory to allocate to virtual machines, however some configurations or levels of load can cause this to occur.
• What to do if threshold is breached:
• Use NUMA aware applications in virtual machines with large amounts of memory.
• Reduce the memory load on the host machine (if live migrating VMs away, prefer any VMs with this issue).
• If one virtual machine is primarily impacted (and performance is less than expected / desired), perform an action that results in that machines physical memory being reallocated to somewhere where there is sufficient memory available for the virtual machine's memory and NUMA configuration. These actions may be live migrate that VM to a different host or power off & power on that VM.
• Avoid using dynamic memory for large memory virtual machines (> 16 GB). Virtual machines with dynamic memory enabled present a single NUMA node to virtual machines (Hyper-V servers running 2012 R2 or earlier). This may or may not impact the performance of the virtual machine.

   
As usual for any of my blog posts - if you have any feedback about any of the above, please provide it - that's how I learn.

• Why: Latency to the Virtual Machine data from Hyper-V is the most common cause of non-optimal virtual machine performance. It significantly helps to know what the normal range is in an specific environment before troubleshooting. This is the first (and for performance – the most important) storage counter to look at.
• Counters:
  • For CSV Disks containing virtual machine data:
    • Cluster CSV File System(*)\Read Latency
    • Cluster CSV File System(*)\Write Latency
  • For both CSV or SMB Storage:
    • SMB Client Shares(*)\Avg. sec/Data Request (only shares containing virtual machine data)
  • When local storage used for VMs (typically non-highly available virtual machines)
    • Logical Disk(*)\Avg. Disk sec/Transfer (only instances containing virtual machine data)

• Threshold (for all of the disk latency thresholds): 0.015 (15ms) is a general threshold, but this will vary - Client virtualization may have higher thresholds (25ms), and server virtualization may have lower thresholds (often 8ms or lower).

• What to do if threshold is exceeded:
  • For non-local storage:
    • Check connection to storage - everything from storage adapters (NIC or FC) to the physical disks, inclusive.
    • Check storage unit load & latency issues
    • Check storage adapter driver & settings (including any load balancing or multipath configuration)
    • Check storage unit and storage path load (both MB/Sec and IOPS) vs baseline numbers.
  • For local storage
    • Check local machine CPU (look for any single cores running 100%), and disk idle time

Disk Idle Time

• Why: This counter provides a clear measurement of what percentage of time the disk remained in idle state, meaning all the requests from the operating system to the disk have been completed and there is zero pending requests. Disk performance (and virtual machine responsiveness) will usually degrade significantly soon after this threshold is reached.
• Counter: Logical Disk(*)\% Idle Time (any instance)
• Threshold: < 50% , sustained > 5 min
• What to do if threshold is exceeded:
  • Check storage unit load & latency issues
  • Check storage adapter driver & settings (including any load balancing or multipath configuration
  • Check storage load (MB/Sec and IOPS) vs baseline numbers.
  • Identify busy virtual machine virtual disks (see "Hyper-V Virtual Storage Device\Normalized Throughput" in this post), and consider redistributing the busy virtual machine virtual disks.

Disk Space

• Why: It is very important to track disks running low on space. When the available space drive containing virtual machine storage falls too low, any operation that involves using more disk space will cause the virtual machine to have an outage and will show a status of "Paused-Critical" (visible in Hyper-V manager). The threshold chosen here should give ample warning before this outage occurs.
• Counter: LogicalDisk(*)\Free Megabytes (any instance other than "_Total" or "C:")
• Threshold: < 200GB (or whatever threshold is appropriate)
  • the value should depend on disk capacity, expected growth, file system type, backup technology and size of regular fluctuations. Suggested values range from 5% (>10TB disks) to 15 % (<1TB disks).
  • If the underlying storage is thinly provisioned, this threshold should generally be 15%.
  • This counter won't show data for remotely accessed SMB storage.
• What to do if threshold is exceeded:
  • Check for snapshots, differencing disks, Hyper-V Replica's (especially initial sync on targets), dynamic disks

Storage MB/sec

• Why: The host counters can be useful information when troubleshooting disk latency and idle time issues. For any shared storage where performance issues are being seen by any device accessing the storage, it is important to identify which host (or combination of hosts) is generating the load. Once the host has been identified, use the virtual machine counters to identify which virtual machine is generating the traffic. If it is an application or service on the host OS, stop it then contact the vendor for a fix or uninstall it - our purpose here is running virtual machines securely, fast and reliably - anything that interferes needs to be carefully evaluated.

• Threshold: no threshold - use this for getting a baseline and for capacity planning.
• Counters (Host):
  • For CSV Disks containing VM data:
    • Cluster CSV File System(*)\IO Read Bytes/sec
    • Cluster CSV File System(*)\IO Write Bytes/sec
  • For both CSV or SMB Storage:
    • SMB Client Shares(*)\Data Bytes/sec (Read Bytes/sec or Write Bytes/sec)
  • When local storage used for virtual machines (typically non-highly available virtual machines)
    • LogicalDisk(*)\Disk Bytes/sec
• Counters (VM):
  • Which virtual machines are generating most storage traffic
    • Hyper-V Virtual Storage Device(*)\Read Bytes/sec
    • Hyper-V Virtual Storage Device(*)\Write Bytes/sec

IOPS(or equivalent):

• Why: This can be useful information when troubleshooting disk latency and idle time issues. For any shared storage where performance issues are being seen by any device accessing the storage, it is important to identify which host (or combination of hosts) is generating the load. Storage teams also like to use these numbers for comparison purposes.

• Threshold: no threshold - use this for getting a baseline and for capacity planning.
• Counter (Host):
  • For CSV Disks containing VM data:
    • Cluster CSV File System(*)\IO Reads/sec
    • Cluster CSV File System(*)\IO Writes/sec
  • For both CSV or SMB Storage:
    • SMB Client Shares(*)\Data Requests/sec (Read Requests/sec / Write Requests/sec)
  • When local storage used for virtual machines (typically non-highly available virtual machines)
    • LogicalDisk(*)\Disk Transfers/sec
• Counters (VM):
  • Which virtual machines are generating most storage traffic
    • Hyper-V Virtual Storage Device(*)\Read Operations/sec
    • Hyper-V Virtual Storage Device(*)\Write Operations/sec

• Counter: Hyper-V Hypervisor Logical Processor(*)\%Total Run Time.
• Threshold: (any instance) > 50% sustained for > 5 min)
• What to do if threshold is exceeded:
  • If _Total meets this, identify sources of CPU load (sounds obvious but.. )
  • If single / small % of CPUs consistently affected, especially (but not necessarily) CPU 0 look for VMQ / RSS config issues, or other hardware issue.
  • If a small number of CPUs are impacted, but the busy CPU does vary, then check the two "% Guest Run Time" processor counters below.

Management operating system CPU usage.

• Counter: Hyper-V Hypervisor Root Virtual Processor(_Total)\% Guest Run Time.
• Threshold (_Total instance) > 5% sustained for > 5 min
• What: This measures the amount of physical CPU time spent running Hyper-V host OS virtual CPUs.
• Why: The host OS doesn't compete for CPU time in Hyper-V. It always immediately gets any CPU time it needs, which is necessary to provide great performance for virtual machines. This means any unnecessary CPU usage in the host OS can have a significant negative performance impact on all virtual machines - which is why the guidance is to install as little as possible on the host OS (absolute minimum of agents, services & applications).
• What to do if threshold is exceeded:
  • Check for any non-default OS running services / agents / processes in this OS, and review if they are really necessary.
  • Check VMQ/RSS configuration (if 10GB NICs are in use) and for bad drivers - especially if this the CPU usage is focused on a small percentage of the hosts OS's VPs.
  • Standard CPU performance troubleshooting - as if it was a physical machine.
  • For general administration, use remote management tools instead of locally or via RDP on the host.

Virtual Machine Processor Usage

• Counter: Hyper-V Hypervisor Virtual Processor(*)\% Guest Run Time.
• Threshold (any instance) > 75%, sustained for > 10min
• What: This measures the amount of physical CPU time each VM virtual CPU instance uses. Each virtual processor can only run on one physical CPU thread at a time.
• Why: Unlike the Processor performance object and task manager inside a virtual machine, this accurately shows how much physical CPU time is being consumed by each VP.
• What to do if threshold is exceeded:
  • Check VMs have sufficient VP (consider adding one of more processors to the VM)
  • Perform standard high CPU usage troubleshooting inside the VM as you would if it was a physical machine to reduce the CPU usage within the VM.

Physical CPU Context Switching

• What: This measures the rate (number of times per second) each logical CPU changes what virtual processor it is running.
• Counter: Hyper-V Hypervisor Logical Processor(*)\Context Switches/sec.
• Threshold: (any instance except for "_Total") > 20000, (sustained for > 5min).
• Why: We use this as a general health & performance indicator for the host & virtual machines. This counter must be used in context with all other activity based counters (CPU, Disk & Network, latency & throughput).
• What to do if threshold is exceeded:
  • Check VM config (particularly remove / disable any active & busy emulated devices)
  • Check that the VM is using the correct version of the integration components.
  • Check host operating system utilization Root VP CPU usage (host OS utilization) - see the "Hyper-V Hypervisor Root Virtual Processor" counter section for the specifics.
  • Check drivers - particularly network and storage drivers, but other too.
  • Check for significant inconsistency across your hosts - it can indicate significant configuration or load differences.