Windows Server 2019 HyperV strange CPU usage pattern

Question

Windows Server 2019 HyperV strange CPU usage pattern
I've 2x dell PowerEdge upgraded from win2016 to win2019, two socket, 6 core, with hyper-threading enable, total 24 logical CPU shown in task manager.
running 17 windows OS guest VMs,
15 VMs with 4 vCPU and dynamic memory of 4GB-8GB
1 VMs with 2 vCPU and dynamic memory of 4GB-8GB
1 VMs with 8 vCPU and dynamic memory of 4GB-8GB
VMs usage varies greatly, but not really overloaded on average, all VMs on SSD

since OS upgrade, I've noticed this CPU usage pattern,
the CPU usage always stuck on CPU0, CPU1 of 0/2/4/6/8/10 affinity as shown in task manager , perfmon also confirmed the same CPU pattern.
I've have since rebooted the HOST and VM every night to see how OS may be re-allocate CPU resource. but it is always stuck on same CPU 0/2/4/6/8/10.
any idea please?

Answer 1

This is a problem with Hyper-Threading when you upgrade your cluster from 2016 to 2019. You need to make sure to upgrade your VM's configuration version when the VM's are off from 8.0 to 9.0. You may also need to set the VM Processor settings to HwThreadCountPerCore to 0 from 1.
Read more here: https://kevinholman.com/2019/07/22/windows-server-2019-hyper-v-might-only-use-half-the-available-cpus/

In Windows Server 2019, Hyper-V now defaults to a more secure “core scheduler” where previous versions used “classic scheduler” for CPU scheduling and isolation. What this means, is that when you migrate a VM over to Hyper-V on WS2019, you need to update the VM configuration and change some settings, or it will only use a single hardware thread per core.

Get-VMHostSupportedVersion

Get-VM | FT Name, Version

Get-VM | Update-VMVersion

Get-VM | Get-VMProcessor | FT VMName,HwThreadCountPerCore

Get-VM | Set-VMProcessor -HwThreadCountPerCore 0

Answer 2

Task Manager in the management OS doesn't see any virtual machine activity. Whatever process(es) cause that, it's not coming from the virtual machines.
You need to do a lot more digging. Find the process(es) that use up so much CPU time and research them. If you're actively in Task Manager at the time that it happens, then you can go to the Processes tab and sort it by CPU usage.
Performance Monitor will probably be a better tool to figure this out.

Answer 3

We are seeing the exact same thing on our 2019 hyperv hosts. Every other cpu is very low usage. All usage is on the other half. This is not numa node related, as the numa nodes in task manager are top and bottom (numa 0 is the top half, numa 1 is the bottom half)
However, I do know that in task manager with hyper threading, that every other cpu graph is the "hyperthreaded one" and not the "real one". So, 0,2,4 are "real" and 1,3,5 are HTT. But I haven't determined what that means. Are my VMs not utilizing hyperthreading or what?

These VMs were NOT upgraded. They were built right on this 2019 server, and they are version 9.0 and have hwThreadCountPerCore set to 0
Their running NUMA configuration is:
Processors: 16
NUMA Nodes: 1
Sockets: 1
Hardware threads per core: 2

Were any of you guys able to figure out why half our logical processors don't seem to be getting used?