What is Azure VMware Solution?
Azure VMware Solution provides private clouds that contain VMware vSphere clusters built from dedicated bare-metal Azure infrastructure. Azure VMware Solution is available in Azure Commercial and Azure Government. The minimum initial deployment is three hosts, with the option to add more hosts, up to a maximum of 16 hosts per cluster. All provisioned private clouds have VMware vCenter Server, VMware vSAN, VMware vSphere, and VMware NSX. As a result, you can migrate workloads from your on-premises environments, deploy new virtual machines (VMs), and consume Azure services from your private clouds. For information about the SLA, see the Azure service-level agreements page.
Azure VMware Solution is a VMware validated solution with ongoing validation and testing of enhancements and upgrades. Microsoft manages and maintains the private cloud infrastructure and software, allowing you to focus on developing and running workloads in your private clouds to deliver business value.
The diagram shows the adjacency between private clouds and VNets in Azure, Azure services, and on-premises environments. Network access from private clouds to Azure services or VNets provides SLA-driven integration of Azure service endpoints. ExpressRoute Global Reach connects your on-premises environment to your Azure VMware Solution private cloud.
Hosts, clusters, and private clouds
Azure VMware Solution clusters are based upon hyper-converged infrastructure. The following table shows the CPU, memory, disk and network specifications of the host.
| Host Type | CPU (Cores/GHz) | RAM (GB) | vSAN Cache Tier (TB, raw***) | vSAN Capacity Tier (TB, raw***) | Regional availability |
|---|---|---|---|---|---|
| AV36 | Dual Intel Xeon Gold 6140 CPUs (Skylake microarchitecture) with 18 cores/CPU @ 2.3 GHz, Total 36 physical cores (72 logical cores with hyperthreading) | 576 | 3.2 (NVMe) | 15.20 (SSD) | Selected regions (*) |
| AV36P | Dual Intel Xeon Gold 6240 CPUs (Cascade Lake microarchitecture) with 18 cores/CPU @ 2.6 GHz / 3.9 GHz Turbo, Total 36 physical cores (72 logical cores with hyperthreading) | 768 | 1.5 (Intel Cache) | 19.20 (NVMe) | Selected regions (*) |
| AV52 | Dual Intel Xeon Platinum 8270 CPUs (Cascade Lake microarchitecture) with 26 cores/CPU @ 2.7 GHz / 4.0 GHz Turbo, Total 52 physical cores (104 logical cores with hyperthreading) | 1,536 | 1.5 (Intel Cache) | 38.40 (NVMe) | Selected regions (*) |
| AV64 | Dual Intel Xeon Platinum 8370C CPUs (Ice Lake microarchitecture) with 32 cores/CPU @ 2.8 GHz / 3.5 GHz Turbo, Total 64 physical cores (128 logical cores with hyperthreading) | 1,024 | 3.84 (NVMe) | 15.36 (NVMe) | Selected regions (**) |
An Azure VMware Solution cluster requires a minimum number of three hosts. You can only use hosts of the same type in a single Azure VMware Solution private cloud. Hosts used to build or scale clusters come from an isolated pool of hosts. Those hosts passed hardware tests and had all data securely deleted before being added to a cluster.
All the above Host Types have 100 Gbps network interface throughput.
(*) details available via the Azure pricing calculator.
(**) AV64 Prerequisite: An Azure VMware Solution private cloud deployed with AV36, AV36P, or AV52 is required prior to adding AV64.
(***) Raw is based upon International Standard of Units (SI) reported by disk manufacturer. Example: 1 TB Raw = 1000000000000 bytes, space calculated by computer in binary (1TB binary = 1099511627776 bytes binary) would equal 931.3 Gigabytes converted from raw decimal.
You can deploy new or scale existing private clouds through the Azure portal or Azure CLI.
Azure VMware Solution private cloud extension with AV64 node size
The AV64 is a new Azure VMware Solution host SKU, which is available to expand (not to create) the Azure VMware Solution private cloud built with the existing AV36, AV36P, or AV52 SKU. Use the Microsoft documentation to check for availability of the AV64 SKU in the region.
Prerequisite for AV64 usage
See the following prerequisites for AV64 cluster deployment.
An Azure VMware solution private cloud is created using AV36, AV36P, or AV52 in AV64 supported region/AZ.
You need one /23 or three (contiguous or noncontiguous) /25 address blocks for AV64 cluster management.
Supportability for customer scenarios
Customer with existing Azure VMware Solution private cloud: When a customer has a deployed Azure VMware Solution private cloud, they can scale the private cloud by adding a separate AV64 vCenter node cluster to that private cloud. In this scenario, customers should use the following steps:
- Get an AV64 quota approval from Microsoft with the minimum of three nodes. Add other details on the Azure VMware Solution private cloud that you plan to extend using AV64.
- Use an existing Azure VMware Solution add-cluster workflow with AV64 hosts to expand.
Customer plans to create a new Azure VMware Solution private cloud: When a customer wants a new Azure VMware Solution private cloud that can use AV64 SKU but only for expansion. In this case, the customer meets the prerequisite of having an Azure VMware Solution private cloud built with AV36, AV36P, or AV52 SKU. The customer needs to buy a minimum of three nodes of AV36, AV36P, or AV52 SKU before expanding using AV64. For this scenario, use the following steps:
- Get AV36, AV36P, or AV52, and AV64 quota approval from Microsoft with a minimum of three nodes each.
- Create an Azure VMware Solution private cloud using AV36, AV36P, or AV52 SKU.
- Use an existing Azure VMware Solution add-cluster workflow with AV64 hosts to expand.
Azure VMware Solution stretched clusters private cloud: The AV64 SKU isn't supported with Azure VMware Solution stretched clusters private cloud. This means that an AV64-based expansion isn't possible for an Azure VMware Solution stretched clusters private cloud.
[!NOTE]
All traffic from an AV64 host towards a customer network will utilize the IP address of the VMKernel Network Interface 1.
AV64 Cluster vSAN fault domain (FD) design and recommendations
The traditional Azure VMware Solution host clusters don't have explicit vSAN FD configuration. The reasoning is the host allocation logic ensures, within clusters, that no two hosts reside in the same physical fault domain within an Azure region. This feature inherently brings resilience and high availability for storage, which the vSAN FD configuration is supposed to bring. More information on vSAN FD can be found in the VMware documentation.
The Azure VMware Solution AV64 host clusters have an explicit vSAN fault domain (FD) configuration. Azure VMware Solution control plane configures seven vSAN fault domains (FDs) for AV64 clusters. Hosts are balanced evenly across the seven FDs as users scale up the hosts in a cluster from three nodes to 16 nodes. Some Azure regions still support a maximum of five FDs as part of the initial release of the AV64 SKU. Refer to the Azure Region Availability Zone (AZ) to SKU mapping table for more information.
Cluster size recommendation
The Azure VMware Solution minimum vSphere node cluster size supported is three. The vSAN data redundancy is handled by ensuring the minimum cluster size of three hosts are in different vSAN FDs. In a vSAN cluster with three hosts, each in a different FD, should an FD fail (for example, the top of rack switch fails), the vSAN data would be protected. Operations such as object creation (new VM, VMDK, and others) would fail. The same is true of any maintenance activities where an ESXi host is placed into maintenance mode and/or rebooted. To avoid scenarios such as these, the recommendation is to deploy vSAN clusters with a minimum of four ESXi hosts.
AV64 host removal workflow and best practices
Because of the AV64 cluster vSAN fault domain (FD) configuration and need for hosts balanced across all FDs, the host removal from AV64 cluster differs from traditional Azure VMware Solution host clusters with other SKUs.
Currently, a user can select one or more hosts to be removed from the cluster using portal or API. One condition is that a cluster should have a minimum of three hosts. However, an AV64 cluster behaves differently in certain scenarios when AV64 uses vSAN FDs. Any host removal request is checked against potential vSAN FD imbalance. If a host removal request creates an imbalance, the request is rejected with the http 409-Conflict response. The http 409-Conflict response status code indicates a request conflict with the current state of the target resource (hosts).
The following three scenarios show examples of instances that normally error out and demonstrate different methods that can be used to remove hosts without creating a vSAN fault domain (FD) imbalance.
Removing a host creates a vSAN FD imbalance with a difference of hosts between most and least populated FD to be more than one. In the following example users, need to remove one of the hosts from FD 1 before removing hosts from other FDs.
Multiple host removal requests are made at the same time and certain host removals create an imbalance. In this scenario, the Azure VMware Solution control plane removes only hosts, which don't create imbalance. In the following example users can't take both of the hosts from the same FDs unless they're reducing the cluster size to four or lower.
A selected host removal causes less than three active vSAN FDs. This scenario isn't expected to occur given that all AV64 regions have five or seven FDs. While adding hosts, the Azure VMware Solution control plane takes care of adding hosts from all seven FDs evenly. In the following example, users can remove one of the hosts from FD 1, but not from FD 2 or 3.
How to identify the host that can be removed without causing a vSAN FD imbalance: A user can go to the vSphere Client interface to get the current state of vSAN FDs and hosts associated with each of them. This helps to identify hosts (based on the previous examples) that can be removed without affecting the vSAN FD balance and avoid any errors in the removal operation.
AV64 supported RAID configuration
This table provides the list of RAID configuration supported and host requirements in AV64 clusters. The RAID-6 FTT2 and RAID-1 FTT3 policies are supported with the AV64 SKU in some regions. In Azure regions that are currently constrained to five FDs, Microsoft allows customers to use the RAID-5 FTT1 vSAN storage policy for AV64 clusters with six or more nodes to meet the service level agreement (SLA). Refer to the Azure Region Availability Zone (AZ) to SKU mapping table for more information.
| RAID configuration | Failures to tolerate (FTT) | Minimum hosts required |
|---|---|---|
| RAID-1 (Mirroring) Default setting. | 1 | 3 |
| RAID-5 (Erasure Coding) | 1 | 4 |
| RAID-1 (Mirroring) | 2 | 5 |
| RAID-6 (Erasure Coding) | 2 | 6 |
| RAID-1 (Mirroring) | 3 | 7 |
Storage
Azure VMware Solution supports the expansion of datastore capacity beyond what is included with vSAN using Azure storage services, enabling you to expand datastore capacity without scaling the clusters. For more information, see Datastore capacity expansion options.
Networking
Azure VMware Solution offers a private cloud environment accessible from on-premises sites and Azure-based resources. Services such as Azure ExpressRoute, VPN connections, or Azure Virtual WAN deliver the connectivity. However, these services require specific network address ranges and firewall ports for enabling the services.
When you deploy a private cloud, private networks for management, provisioning, and vMotion get created. You use these private networks to access VMware vCenter Server and VMware NSX Manager and virtual machine vMotion or deployment.
ExpressRoute Global Reach is used to connect private clouds to on-premises environments. It connects circuits directly at the Microsoft Edge level. The connection requires a virtual network (vNet) with an ExpressRoute circuit to on-premises in your subscription. The reason is that vNet gateways (ExpressRoute Gateways) can't transit traffic, which means you can attach two circuits to the same gateway, but it doesn't send the traffic from one circuit to the other.
Each Azure VMware Solution environment is its own ExpressRoute region (its own virtual MSEE device), which lets you connect Global Reach to the 'local' peering location. It allows you to connect multiple Azure VMware Solution instances in one region to the same peering location.
Note
For locations where ExpressRoute Global Reach isn't enabled, for example, because of local regulations, you have to build a routing solution using Azure IaaS VMs. For some examples, see Azure Cloud Adoption Framework - Network topology and connectivity for Azure VMware Solution.
Virtual machines deployed on the private cloud are accessible to the internet through the Azure Virtual WAN public IP functionality. For new private clouds, internet access is disabled by default.
For more information, see Networking architecture.
Access and security
Azure VMware Solution private clouds use vSphere role-based access control for enhanced security. You can integrate vSphere SSO LDAP capabilities with Microsoft Entra ID. For more information, see the Access and identity architecture page.
vSAN data-at-rest encryption, by default, is enabled and is used to provide vSAN datastore security. For more information, see Storage architecture.
Data residency and customer data
Azure VMware Solution doesn't store customer data.
VMware software versions
The VMware solution software versions used in new deployments of Azure VMware Solution private clouds are:
| Software | Version |
|---|---|
| VMware vCenter Server | 8.0 U2d |
| VMware ESXi | 8.0 U2b |
| VMware vSAN | 8.0 U2 |
| VMware vSAN on-disk format | 19 |
| VMware vSAN storage architecture | OSA |
| VMware NSX | 4.1.1 |
| VMware HCX | 4.9.1 |
| VMware Site Recovery Manager | 8.8.0.3 |
| VMware vSphere Replication | 8.8.0.3 |
The current running software version is applied to new clusters added to an existing private cloud, if the vCenter Server version supports it.
Host and software lifecycle maintenance
Regular upgrades of the Azure VMware Solution private cloud and VMware software ensure the latest security, stability, and feature sets are running in your private clouds. For more information, see Host maintenance and lifecycle management.
Monitoring your private cloud
Once you deployed Azure VMware Solution into your subscription, Azure Monitor logs are generated automatically.
In your private cloud, you can:
- Collect logs on each of your VMs.
- Download and install the MMA agent on Linux and Windows VMs.
- Enable the Azure diagnostics extension.
- Create and run new queries.
- Run the same queries you usually run on your VMs.
Monitoring patterns inside the Azure VMware Solution are similar to Azure VMs within the IaaS platform. For more information and how-tos, see Monitoring Azure VMs with Azure Monitor.
Customer communication
You can find service issues, planned maintenance, health advisories, and security advisories notifications published through Service Health in the Azure portal. You can take timely actions when you set up activity log alerts for these notifications. For more information, see Create Service Health alerts using the Azure portal.
Azure VMware Solution responsibility matrix - Microsoft vs customer
Azure VMware Solution implements a shared responsibility model that defines distinct roles and responsibilities of the two parties involved in the offering: customer and Microsoft. The shared role responsibilities are illustrated in more detail in the following two tables.
The shared responsibility matrix table outlines the main tasks that customers and Microsoft each handle in deploying and managing both the private cloud and customer application workloads.
The following table provides a detailed list of roles and responsibilities between the customer and Microsoft, which encompasses the most frequent tasks and definitions. For further questions, contact Microsoft.
| Role | Task/details |
|---|---|
| Microsoft - Azure VMware Solution | Physical infrastructure
(optional) VMware HCX deploys with fully configured compute profile on cloud side as add-on (optional) VMware SRM deploys, upgrade, and scale up/down Support - Private cloud platforms and VMware HCX |
| Customer | Request Azure VMware Solution host quote with Microsoft Plan and create a request for private clouds on Azure portal with:
Add or delete hosts requests to cluster from Portal Deploy/lifecycle management of partner (third party) solutions |
| Partner ecosystem | Support for their product/solution. For reference, the following are some of the supported Azure VMware Solution partner solution/product:
|
Next steps
The next step is to learn key private cloud architecture concepts.