Standards Compliant Virtual Switch

 As I mentioned in my previous post, the main goal in building VMSWITCH was to build a standards compliant, high performance virtual switch. There are many ways to interpret standards compliant because there are many standards. So to be specific, our goal was to build packet forwarding based on 802.1q and mimic physical network semantics (such as link up/down) as much as possible.  This was to make sure that whatever works in a physical network, works in the virtual environment as well. We defined three main objects in VMSWITCH, vSwitch, vPort and NIC. A vSwitch is an instance of a virtual switch that provides packet forwarding and various other features provided by a switch such as QoS, ACL etc. A vPort is analogous to a physical switch port and has configuration associated with it for various features. And finally, a NIC objects, that acts as the endpoint connecting to a vPort. This is similar to the physical network, where a host has a physical NIC, which connects to a physical port on a physical switch.

The clear separation of these objects, allowed us to model our switch design and code independent of the type of network device we virtualize e.g. legacy NIC versus virtual NIC as well as provide the support network semantics similar to physical network. For example, by disconnecting a NIC from a vPort, we could generate link down notification. It also gave us the flexibility to support various types of vSwitch (discussed later) without requiring any changes in our core VMSWITCH implementation.

The picture below shows an example setup, multiple vSwitch are created with different number of vPort on each one of them and vPort(s) connecting to different types of NIC(s).

This flexibility allowed creation of different types of vSwitch in the Hyper-V UI. If you use Hyper-V UI, you would notice that vSwitch creation page allows three different types of vSwitches, External, Internal and Private vSwitch. The external switch provides connectivity via a physical NIC, internal switch only provides connectivity to root partition via host virtual NIC and private switch only provides inter-VM connectivity. The key here is that, in the VMSWITCH implementation, there is no type of vSwitch, each vSwitch is the same. The UI classifies the vSwitch, based on what types of NICs are connected to the vPort(s) of that vSwitch. This kept VMSWITCH design simpler but allowed us to provide a friendlier UX for our customers.

Control and Data Plane

The control interface to VMSWITCH is via IOCTL, which is standard mechanism in Windows to interface user mode applications with kernel mode. The management application invokes various IOCTLs to carry out operations such as creation of a vSwitch, connecting a vPort to a NIC etc.

The data plane interface is specific to each type of NIC that connects to a vPort. The overall control and data plane interactions are shown in the picture below.

This picture shows a vSwitch with 4 vPort, and each vPort connects to a different type of NIC. The control plane interaction happens via IOCTL. The picture also shows the internal representation of various NICs in VMSWITCH.  The data transfer happens via interface specific to the type of NIC. 

A physical NIC connected to VMSWITCH is represented as NDIS protocol and does packet I/O via NDIS interfaces defined for a protocol driver. The host vNIC is represented by an NDIS Miniport in VMSWITCH and does packet I/O via NDIS interfaces defined for a miniport driver. The legacy NIC is supported by emulation running in VMWP process and is represented in VMSWITCH as direct I/O NIC. It uses IRP_MJ_READ and IRP_MJ_WRITE to do packet I/O with VMSWITCH. Lastly, a virtual NIC in the VM is represented as VM NIC in VMSWITCH and uses VMBUS interfaces for packet I/O.

Core Packet Processing

The core VMSWITCH packet processing logic has three layers, Ingress, Forwarding and Egress. In the initial version, we called it Source, Route and Destination. In the source processing step, we applied various source vPort specific policies on the packet and if the packet was not dropped, then we did the packet forwarding processing and once we computed the destination vPort list of the packet, for each destination, we applied destination vPort specific policies. Packets that did not get dropped based on these policies were delivered to their destination vPort(s).

All NIC entry points to VMSWITCH do the interface specific processing on the packets and convert the packets from interface specific format to a common format that is fed to VMSWITCH packet processing pipeline. The VMSWITCH packet processing code is largely independent of the type of NIC, except for cases where offload handling is needed, since offloads are specific to the type of NIC. The modularity of this architecture made is easy for us to move VMSWITCH to Extensible VMSWITCH during Windows Server 2012 release with minimal changes to our core packet processing logic.

Performance

The packet processing pipeline of VMSWITCH is largely lock free, allowing for parallel processing of packets. By using features such as VMQ, VMSWITCH is able to utilize any available CPU for packet processing and carry those out without requiring serialization as much as possible. Most of the operations are lock free and where lock is needed, they are either held for short duration or interlocked operations are used. As an example of lock free operation, VMSWITCH maintains various packet counters on vSwitch, vPort and NIC objects. Updating to these counters atomically can add significant CPU cycles to the overall packet processing cost. To reduce (or almost eliminate) this overhead, per processor packet counters are used that are updated at DISPATCH_LEVEL thus providing lock free updates to these. A per processor counter design makes querying of packet counters less efficient, because now we have to add all per-processor counters. However, since querying of counters is not very frequent as compared to updating the counters on data path, the optimization helps in the critical path i.e. the core packet processing. Similar techniques are used to make sure that overall packet processing is highly parallel and efficient. This coupled with offload support is able to provide performance that is needed by some of the most demanding workloads running in VMs.

This is all I have for today, in future posts I would talk about VMSWITCH offload support, Extensibility model and more.

The goal was simple (well it sounded simple), build a compliant, best in class, high performance network device virtualization. It was my first project to build something in kernel mode, it was exciting, and challenging. We spent countless hours on the whiteboard debating how the overall solution should look like, what should each component do, what would be the right layering, right modules and all those fun parts of the project. And this is what we built eventually.

First and foremost, we decided that we would build a switch that is 802.1Q compliant and behaves like a physical switch so that the VMs (or child partitions) running on Hyper-V would have exactly same network semantics as a physical machine. This makes sure that virtualization doesn't create a bar to adoption and what works in physical environment, would work in a virtual environment. This is the component VMSWITCH shown above. I will talk more about it in a future post.

Next thing we decided was to create two separate device types, one that emulated a real hardware, in our case Intel DEC 21140A, shown as "Legacy NIC" above and other than ran over a virtual bus (VMBUS), shown as "Virtual NIC" above. Legacy NIC was to provide compatibility with existing operating systems since most of them had a device driver for it already, whereas Virtual NIC was to provide high speed networking for "enlightened" operating systems for which we decided to build our virtual bus.

Legacy NIC is built using device emulation, such that to the operating system running in the VM, it seems like a real physical device and accessed via the bus and device registers. The access to these registers is intercepted by the hypervisor, that are then redirected to a user mode process VMWP in the root partition, which carries over the emulation. It then sends the network packet to VMSWITCH, which does the forwarding. Similarly, when VMSWITCH receives a packet destined for the legacy NIC, it is given to VMWP, which indicates it to the child partition by issuing an interrupt via the legacy NIC device emulation. This, is a slow process due to multiple context switches between, child partition, hypervisor and root partition. However this allows networking to work on "unenlightend" or legacy operating system on Hyper-V.

Virtual NIC is built over the virtual bus and is virtualization aware. Being virtualization aware, it doesn't go through traditional device access mechanisms and uses shared memory to transfer data between child partition and root partition. Virtual NIC makes it easier to provide various NIC offloads to the child partition and since we don't have to fiddle around with emulating device registers for every new offload, we can enable new offloads easily. I would talk about, how some of the offloads work on Hyper-V in a future post.

The picture below (courtesy: https://msdn.microsoft.com/en-us/library/dd722833(v=bts.10).aspx) does a good job in showing the difference between enlightened and unenlightend operating system on Hyper-V. 

So this is how network device virtualization works from a 1000 ft level on Hyper-V. In future post, I would go in the details of how VMSWITCH works, how offload support and various other features work. My plan is to finish the basic device virtualization series and then switch to network virtualization and SDN, and describe the internals of some of the cool stuff we are doing in Hyper-V.

Thanks for reading, I hope you find it useful. Please feel free to let me know if you would like me to talk about a particular area or specific topic.