The IaaS PLA Fabric Architecture guide covers common elements in hybrid cloud fabric design: compute, network, storage and virtualization.  It defines clear design patterns for customers to follow when building hybrid cloud fabrics using Windows Server and Microsoft Azure.

The IaaS PLA Fabric Management Architecture guide covers the architecture and capabilities provided by the System Center components focused on supporting IaaS solutions including VMM, Operations Manager, Service Manager, Orchestrator, App Controller, SPF, SMA, Service Reporting and Windows Azure Pack.

The IaaS PLA Deployment guide covers a comprehensive end-to-end solution-level deployment of System Center using the aforementioned components. We have several previously published releases of the IaaS PLA guides which support Windows Server 2012, System Center 2012 and System Center 2012 SP1.  The latest release wave supports Windows Server 2012 R2, System Center 2012 R2 and Microsoft Azure.

PLAs:

• Provide a technology definition that helps you understand the capability and how it is supported by Windows Server, System Center or Microsoft Azure
• Help you arrive at a design based on your needs through the use of clearly defined design patterns

You can download the PLAs today from the TechNet Gallery:

• Infrastructure as a Service Product Line Architecture-Fabric Architecture Guide
• Infrastructure as a Service Product Line Architecture - Fabric Management Guide
• Infrastructure as a Service Product Line Architecture-Deployment Guide
• Infrastructure as a Service Product Line Architecture - Consolidated Set

Let us know what you think of the PLAs! You’re welcome to write to me at tomsh@microsoft.com and I’ll get your feedback to the right people.

HTH,

Tom

Tom Shinder
tomsh@microsoft.com
Knowledge Engineer, DDEC Solutions Group
Follow me on Twitter: https://twitter.com/tshinder
Facebook: https://www.facebook.com/tshinder

Prepared by

Adam Fazio, David Ziembicki, Joel Yoker, Artem Pronichkin, Jeff Baker, Michael Lubanski, Robert Larson, Steve Chadly, Alex Lee, Yuri Diogenes, Carlos Mayol Berral, Ricardo Machado, Sacha Narinx, Tom Shinder, Jim Dial

We highly recommend that you download the Word document as it has the latest revisions to the guide and also contains the diagrams. This web version does not include the diagrams that are inside of the Word document.

Table of Contents

1 Introduction. 9

1.1 Scope. 9

1.2 Microsoft Services. 9

2 IaaS Product Line Architecture Overview.. 11

2.1 IaaS Reference Architectures. 11

2.1.1 Product Line Architecture Fabric Design Patterns. 12

2.2 Windows Hardware Certification.. 14

2.3 Windows Licensing.. 16

3 Software Defined Infrastructure Pattern Overview.. 17

4 Non-Converged Infrastructure Pattern Overview.. 21

5 Converged Infrastructure Pattern Overview.. 23

6 Hybrid Infrastructure Pattern Overview.. 25

7 Storage Architecture. 27

7.1 Drive Architectures. 27

7.1.1 Serial ATA (SATA) 27

7.1.2 SAS. 27

7.1.3 Nearline SAS (NL-SAS) 29

7.1.4 Fibre Channel 29

7.1.5 Solid-State Storage. 29

7.1.6 Hybrid Drives. 30

7.1.7 Advanced Format (4K) Disk Compatibility. 30

7.2 Storage Controller Architectures. 30

7.2.1 SATA III 31

7.2.2 PCIe/SAS HBA.. 32

7.2.3 PCIe RAID/Clustered RAID.. 32

7.2.4 Fibre Channel HBA.. 33

7.3 Storage Networking.. 34

7.3.1 Fibre Channel 34

7.3.2 iSCSI 34

7.3.3 Fibre Channel over Ethernet (FCoE) 35

7.3.4 InfiniBand. 35

7.3.5 Switched SAS. 36

7.3.6 Network File System.. 37

7.3.7 SMB 3.0. 37

7.4 File Services. 43

7.4.1 Storage Spaces. 43

7.4.2 Resilient File System.. 46

7.4.3 NTFS Improvements. 46

7.4.4 Scale-Out File Server Cluster Architecture. 47

7.5 Storage Features. 49

7.5.1 Data Deduplication. 49

7.5.2 Thin Provisioning and Trim.. 50

7.5.3 Volume Cloning. 50

7.5.4 Volume Snapshot 50

7.5.5 Storage Tiering. 51

7.6 Storage Management and Automation.. 51

7.6.1 Windows Offloaded Data Transfers (ODX) 52

8 Network Architecture. 53

8.1 Network Architecture Patterns. 53

8.1.1 Hierarchical 53

8.1.2 Flat Network. 54

8.1.3 Network Virtualization (Software-Defined Networking) 54

8.2 Network Performance and Low Latency. 55

8.2.1 Data Center Bridging. 55

8.2.2 Virtual Machine Queue (VMQ) 55

8.2.3 IPsec Task Offload. 56

8.2.4 Quality of Service (QoS) 56

8.2.5 Remote Direct Memory Access and SMB Direct 57

8.2.6 Receive Segment Coalescing. 57

8.2.7 Receive-Side Scaling. 57

8.2.8 SR-IOV. 58

8.2.9 TCP Chimney Offload. 59

8.3 Network High Availability and Resiliency. 59

8.3.1 NIC Teaming. 59

8.4 Network Isolation and Security. 62

8.4.1 VLANs. 63

8.4.2 Trunk Mode to Virtual Machines. 63

8.4.3 Private VLANs. 63

8.4.4 ARP and Neighbor Discovery Spoofing Protection. 64

8.4.5 Router Guard. 64

8.4.6 DHCP Guard. 65

8.4.7 Port ACLs. 65

8.4.8 Network Virtualization. 66

9 Compute Architecture. 69

9.1 Server Architecture. 69

9.1.1 Server and Blade Network Connectivity. 69

9.1.2 Microsoft Multipath I/O.. 69

9.1.3 Consistent Device Naming (CDN) 70

9.2 Failover Clustering.. 70

9.2.1 Cluster-Aware Updating (CAU) 70

9.2.2 Cluster Shared Volume (CSV) 71

9.3 Hyper-V Failover Clustering.. 76

9.3.1 Host Failover-Cluster Topology. 76

9.3.2 Cluster Quorum and Witness Configurations. 76

9.3.3 Host Cluster Networks. 78

9.3.4 Hyper-V Application Monitoring. 80

9.3.5 Virtual Machine Failover Prioritization. 80

9.3.6 Virtual Machine Anti-Affinity. 80

9.3.7 Virtual Machine Drain on Shutdown. 80

9.3.8 Shared Virtual Hard Disk. 81

10 Hyper-V Virtualization Architecture. 82

10.1 Hyper-V Features. 82

10.1.1 Hyper-V Host and Guest Scale-Up. 82

10.1.2 Hyper-V over SMB 3.0. 82

10.1.3 Virtual Machine Mobility. 82

10.1.4 Storage Migration. 84

10.1.5 Hyper-V Virtual Switch. 84

10.1.6 Virtual Fibre Channel 87

10.1.7 VHDX. 88

10.1.8 Guest Non-Uniform Memory Access (NUMA) 89

10.1.9 Dynamic Memory. 89

10.1.10 Hyper-V Replica. 90

10.1.11 Resource Metering. 91

10.1.12 Enhanced Session Mode. 91

10.2 Hyper-V Guest Virtual Machine Design.. 92

10.2.1 Virtual Machine Storage. 92

10.2.2 Virtual Machine Networking. 94

10.2.3 Virtual Machine Compute. 94

10.2.4 Linux-Based Virtual Machines. 96

10.2.5 Automatic Virtual Machine Activation. 97

11 Microsoft Azure and IaaS Architecture. 98

11.1 Microsoft Azure Services. 99

11.1.1 Microsoft Azure Compute Services. 99

11.1.2 Microsoft Azure Data Services. 99

11.1.3 Microsoft Azure Network Services. 100

11.1.4 Microsoft Azure Application Services. 100

11.2 Microsoft Azure Accounts and Subscriptions. 102

11.2.1 Sharing Service Management by Adding Co-Administrators. 103

11.2.2 Manage Storage Accounts for Your Subscription. 104

11.2.3 Create Affinity Groups to Use with Storage Accounts and Hosted Services 104

11.2.4 Add Management Certificates to a Microsoft Azure Subscription. 104

11.2.5 Creating and Managing Microsoft Azure Environments. 104

11.3 Microsoft Azure Service-Level Agreements. 106

11.3.1 Caching. 106

11.3.2 Content Delivery Network (CDN) 106

11.3.3 Cloud Services, Virtual Machines, and Virtual Network. 106

11.3.4 Media Services. 107

11.3.5 Mobile Services. 107

11.3.6 Multi-Factor Authentication. 107

11.3.7 Service Bus. 107

11.3.8 SQL Server Database. 108

11.3.9 SQL Server Reporting Services. 108

11.3.10 Storage. 108

11.3.11 Web Sites. 108

11.4 Microsoft Azure Pricing.. 109

11.5 Extending the Data Center Fabric to Microsoft Azure. 109

11.5.1 Data Management Services in Microsoft Azure. 109

11.5.2 Microsoft Azure and IaaS. 120

11.5.3 Networking in Microsoft Azure. 131

12 Fabric and Fabric Management 141

12.1 Fabric. 141

12.2 Fabric Management 142

12.2.1 Fabric Management Host Architecture. 142

13 Non-Converged Architecture Pattern. 143

13.1 Compute. 144

13.1.1 Hyper-V Host Infrastructure. 144

13.2 Network. 146

13.2.1 Host Connectivity. 146

13.3 Storage. 148

13.3.1 Storage Connectivity. 148

13.3.2 Storage Infrastructure. 149

14 Converged Architecture Pattern. 152

14.1 Compute. 153

14.1.1 Hyper-V Host Infrastructure. 153

14.2 Network. 154

14.2.1 Host Connectivity. 154

14.3 Storage. 155

14.3.1 Storage Connectivity. 155

14.3.2 Storage Infrastructure. 156

15 Software Defined Infrastructure Architecture Pattern. 159

15.1 Compute. 160

15.1.1 Hyper-V Host Infrastructure. 160

15.2 Network. 162

15.2.1 Host Connectivity. 162

15.3 Storage. 163

15.3.1 Storage Connectivity. 163

15.3.2 Scale-Out File Server Cluster Architecture. 163

15.3.3 Storage Infrastructure. 167

16 Multi-Tenant Designs. 169

16.1 Requirements Gathering.. 169

16.2 Infrastructure Requirements. 170

16.3 Multi-Tenant Storage Considerations. 171

16.3.1 SMB 3.0. 171

16.4 Multi-Tenant Network Considerations. 174

16.4.1 Windows Network Virtualization. 174

16.4.2 Hyper-V Virtual Switch. 177

16.4.3 Example Network Design. 178

16.5 Multi-Tenant Compute Considerations. 180

16.5.1 Hyper-V Role. 181

16.5.2 Failover Clustering. 182

16.5.3 Resource Metering. 182

16.5.4 Management 183

1 Introduction

The goal of the Infrastructure-as-a-Service (IaaS) product line architecture (PLA) is to help organizations develop and implement private cloud infrastructures quickly while reducing complexity and risk. The IaaS PLA provides a reference architecture that combines Microsoft software, consolidated guidance, and validated configurations with partner technologies such as compute, network, and storage architectures, in addition to value-added software features.

The private cloud model provides much of the efficiency and agility of cloud computing, with the increased control and customization that are achieved through dedicated private resources. By implementing private cloud configurations that align to the IaaS PLA, Microsoft and its hardware partners can help provide organizations the control and the flexibility that are required to reap the potential benefits of the private cloud.

The IaaS PLA utilizes the core capabilities of the Windows Server operating system, Hyper-V, and System Center to deliver a private cloud infrastructure as a service offering. These are also the key software features and components that are used for every reference implementation.

1.1 Scope

The scope of this document is to provide customers with the necessary guidance to develop solutions for a Microsoft private cloud infrastructure in accordance with the IaaS PLA patterns that are identified for use with the Windows Server 2012 R2 operating system. This document provides specific guidance for developing fabric architectures (compute, network, storage, and virtualization layers) for an overall private cloud solution. Guidance is provided for the development of an accompanying fabric management architecture that uses System Center 2012 R2.

1.2 Microsoft Services

This PLA document was developed by Microsoft Services. Microsoft Services is comprised of a global team of architects, engineers, consultants, and support professionals who are dedicated to helping customers maximize the value of their investment in Microsoft software. Microsoft Services supports customers in over 82 countries, helping them plan, deploy, support, and optimize Microsoft technologies. Microsoft Services works closely with Microsoft Partners by sharing their technological expertise, solutions, and product knowledge. For more information about the solutions that Microsoft Services offers or to learn about how to engage with Microsoft Services and Microsoft Partners, please visit the Microsoft Services website.

2 IaaS Product Line Architecture Overview

The IaaS PLA is focused on deploying virtualization fabric and fabric management technologies in Windows Server and System Center to support private cloud scenarios. This PLA includes reference architectures, best practices, and processes for streamlining deployment of these platforms to support private cloud scenarios.

This part of the IaaS PLA focuses on delivering core foundational virtualization fabric infrastructure guidance that aligns to the defined architectural patterns within this and other Windows Server 2012 R2 private cloud programs. The resulting Hyper-V infrastructure in Windows Server 2012 R2 can be leveraged to host advanced workloads. The accompanying Fabric Management Architecture Guide, contains fabric management scenarios that use System Center components. Scenarios that are relevant to this release include:

· Resilient infrastructure: Maximize the availability of IT infrastructure through cost-effective redundant systems that prevent downtime, whether planned or unplanned.

· Centralized IT: Create pooled resources with a highly virtualized infrastructure that support maintaining individual tenant rights and service levels.

· Consolidation and migration: Remove legacy systems and move workloads to a scalable high-performance infrastructure.

· Preparation for the cloud: Create the foundational infrastructure to begin the transition to a private cloud solution.

2.1 IaaS Reference Architectures

Microsoft Private Cloud programs have two main solutions as shown in Figure 1. This document focuses on the open solutions model, which can be used to service the enterprise and hosting service provider audiences.

Figure 1 Branches of the Microsoft Private Cloud

Small- or medium-sized enterprises should plan a reference architecture that defines the requirements that are necessary to design, build, and deliver virtualization and private cloud solutions, including hosting service provider implementations.

Figure 2 shows examples of these reference architectures.

Figure 2 Examples of reference architectures

Each reference architecture combines concise guidance with validated configurations for the compute, network, storage, and virtualization layers. Each architecture presents multiple design patterns to enable the architecture, and each design pattern describes the minimum requirements for each solution.

2.1.1 Product Line Architecture Fabric Design Patterns

As previously described, Windows Server 2012 R2 utilizes innovative hardware capabilities, and it enables what were previously considered advanced scenarios and capabilities from commodity hardware. These capabilities have been summarized into initial design patterns for the IaaS PLA. Identified patterns include the following infrastructures:

· Software-defined infrastructure

· Non-converged infrastructure

· Converged infrastructure

Each design pattern guide outlines the high-level architecture, provides an overview of the scenario, identifies technical requirements, outlines all dependencies, and provides guidelines as to how the architectural guidance applies to each deployment pattern. Each pattern also includes an array of fabric constructs in the categories of compute, network, storage, and virtualization. Each pattern is outlined in this guide with an overview of the pattern and a summary of how each pattern leverages each feature area.

The following features are common across each of the design patterns:

Required Features	Optional Features
· Dedicated fabric management hosts	· Addition of single root I/O virtualization (SR-IOV) network interface cards
· 10 gigabit Ethernet (GbE) or higher network connectivity	· Addition of a certified Hyper-V extensible virtual switch extension
· Redundant paths for all storage networking components (such as redundant serial attached SCSI (SAS) paths, and Multipath I/O (MPIO) for Fibre Channel and SMB Multichannel where appropriate)
· SMI-S or SMP–compliant management interfaces for storage components
· Remote direct memory access (RDMA) network connectivity (RoCE, InfiniBand or iWARP)
· Shared storage

The following table outlines the Windows Server 2012 R2 features and technologies that are common to all patterns:

Windows Server 2012 R2 Feature	Key Scenarios
Increased Virtual Processor to Logical Processor ratio	Removal of previous limits of 8:1 processor ratios for server workloads and 12:1 processor ratios for client workloads.
Increased virtual memory and Dynamic Memory	Supports up to 1 TB of memory inside virtual machines.
Virtual machine guest clustering enhancements	Supports virtual machine guest clusters by using a shared virtual hard disk, iSCSI connections, or the Hyper-V Fibre Channel adapter to connect virtual machines to shared storage.
Hyper-V virtual switch	A virtual Ethernet switch that allows filtering, capturing, and forwarding extensions that are to be added by non-Microsoft vendors to support additional virtual-switch functionality on the Hyper-V platform.
Cluster-aware updating	Provides the ability to apply updates to running failover clusters through coordinated patching of individual failover-cluster nodes.
Live migration enhancements	Supports migrating virtual machines without using shared storage, memory compression, or the SMB 3.0 protocol
Support for SR-IOV	Provides the ability to assign a network adapter that supports single-root I/O virtualization (SR-IOV) directly to a virtual machine.
Support for 4K physical disks	Supports native 4K disk drives on hosts.
Diskless network boot with iSCSI  Target Server	Provides the network-boot capability on commodity hardware by using an iSCSI boot–capable network adapter or a software boot loader.
Virtual machine generation enhancements	Windows Server 2012 R2 introduces Generation 2 virtual machines, which support new functionality on virtual machines such as UEFI firmware, PXE boot, and Secure Boot.
Virtual machine storage enhancements (VHDX format)	Supports VHDX format for disks that are up to 64 TB in size and shared virtual hard disks
Windows NIC Teaming	Supports switch-independent and switch-dependent load distribution by using physical and virtual network connections.
Data Center Bridging	Provides hardware support for converged fabrics, which allows bandwidth allocation and priority flow control.

Table 1 Windows Server 2012 R2 features and key scenarios applicable to all patterns

2.2 Windows Hardware Certification

In IaaS PLA implementations, it is mandatory that each architecture solution pass the following validation requirements:

· Windows hardware certification

· Failover-clustering validation

· Clustered RAID controller validation (if a non-Microsoft clustered RAID controller is used)

These rule sets are described in the following subsections.

2.2.1.1 Windows Hardware Certification

Hardware solutions must receive validation through the Microsoft “Certified for Windows Server 2012 R2” program before they can be presented in the Windows Server Catalog. The catalog contains all servers, storage, and other hardware devices that are certified for use with Windows Server 2012 and Hyper-V.

The Certified for Windows Server 2012 R2 logo demonstrates that a server system meets the high technical bar set by Microsoft for security, reliability, and manageability, and any required hardware components that support all of the roles, features, and interfaces that Windows Server 2012 R2 supports.

The logo program and the support policy for failover-clustering solutions require that all the individual components that make up a cluster configuration earn the appropriate "Certified for" or "Supported on” Windows Server 2012 R2 designations before they are listed in their device-specific categories in the Windows Server Catalog.

For more information, open the Windows Server Catalog. Under Hardware Testing Status, click Certified for Windows Server 2012 R2. The two primary entry points for starting the logo-certification process are Windows Hardware Certification Kit (HCK) downloads and the Windows Dev Center Hardware and Desktop Dashboard.

Validation requirements include failover-clustering validation and clustered RAID controller validation, as described in the following subsections.

2.2.1.2 Failover-Clustering Validation

For Windows Server 2012 R2, failover clustering can be validated by using the Cluster Validation Tool to confirm network and shared storage connectivity between the nodes of the cluster. The tool runs a set of focused tests on the servers that are to be used as nodes in a cluster, or are already members of a given cluster. This failover-cluster validation process tests the underlying hardware and software directly, and individually obtains an accurate assessment of whether the failover cluster has the ability to support a given configuration.

Cluster validation is used to identify hardware or configuration issues before the cluster enters production. This helps make sure that a solution is truly dependable.

In addition, cluster validation can be performed as a diagnostic tool on configured failover clusters. Failover clusters must be tested and they must pass the failover-cluster validation to receive customer support from Microsoft Customer Support Services (CSS).

2.2.1.3 Clustered RAID Controller Validation

Clustered RAID controllers are a relatively new type of storage interface card that can be used with shared storage and cluster scenarios. RAID controllers that are set up across configured servers provide shared storage, and the clustered RAID controller solution must pass the clustered RAID controller validation.

If the solution includes a clustered RAID controller, this validation requirement includes Windows Licensing.

2.3 Windows Licensing

IaaS PLA architectures use the Windows Server 2012 R2 Standard or Windows Server 2012 R2 Datacenter.

The packaging and licensing for Windows Server 2012 R2 have been updated to simplify purchasing and reduce management requirements, as shown in the following table. The Windows Server 2012 R2 Standard and Datacenter editions are differentiated only by virtualization rights—two virtual instances for the Standard edition, and an unlimited number of virtual instances for the Datacenter edition.

For more information about Windows Server 2012 R2 licensing, see the Windows Server 2012 R2 Datasheet or Windows Server 2012: How to Buy.

For information about licensing in virtual environments, see Microsoft Volume Licensing Brief: Licensing Microsoft Server Products in Virtual Environments.

3 Software Defined Infrastructure Pattern Overview

The Software Defined Infrastructure pattern (previously referred to as Continuous Availability over Server Message Block (SMB) Storage) supports deployments in Windows Server 2012 R2 that use Hyper-V and Failover Clustering. Continuous availability and transparent failover are delivered over a Scale-Out File Server cluster infrastructure, and SMB shared storage is provided by using a converged hardware configuration and native capabilities in the Windows Server 2012 R2 operating system. This pattern has three variations:

· Variation A: SMB Direct using shared serial attached SCSI (SAS) and Storage Spaces

· Variation B: SMB Direct using storage area network (SAN)

· Variation C: SMB 3.0-enabled storage

Note   SMB Direct is based on SMB 3.0, and it supports the use of network adapters that have remote direct memory access (RDMA) capability.

Variation A uses SMB Direct using shared SAS and Storage Spaces to provide storage capabilities over direct-attached storage technologies. This pattern combines a Scale-Out File Server cluster infrastructure with SMB Direct to provide back-end storage that has similar characteristics to traditional SAN infrastructures and supports Hyper-V and SQL Server workloads.

Figure 3 outlines a conceptual view of Variation A.

Figure 3 Conceptual view of variation A

Variation B uses SMB Direct with SAN-based storage, which provides the advanced storage capabilities that are found in storage area network (SAN) infrastructures. SAN-based storage solutions typically provide additional features beyond what can be provided natively through the Windows Server 2012 R2 operating system by using shared direct-attached “Just a Bunch of Drives” (JBOD) storage technologies. Although this variation is generally more expensive, its primary trade-offs weigh capability and manageability over cost.

Variation B is similar to Variation A. It utilizes a Scale-Out File Server cluster infrastructure with SMB Direct; however, the back-end storage infrastructure is a SAN-based storage array. In this variation, innovative storage capabilities that are typically associated with SAN infrastructures can be utilized in conjunction with RDMA and SMB connectivity for Hyper-V workloads.

Figure 4 outlines a conceptual view of Variation B.

Figure 4 Conceptual view of variation B

In Variation C, instead of using Scale-Out File Server clusters and SMB Direct, SMB 3.0-enabled storage devices are used to provide basic storage capabilities, and Hyper-V workloads utilize the SMB shared resources directly. This configuration might not provide advanced storage capabilities, but it provides an affordable storage option for Hyper-V workloads.

Figure 5 outlines a conceptual view of Variation C.

Figure 5 Conceptual view of variation C

Although the following list of requirements is not comprehensive, the Software Defined Infrastructure pattern requires the following features:

· All the common features listed earlier

· Dedicated hosts for a Scale-Out File Server cluster (for Variations A and B)

· Shared SAS JBOD storage array (required for Variation A)

· SMB 3.0-enabled storage array (optional for Variation C only)

Table 3 outlines Windows Server 2012 R2 features and technologies that are utilized in this architectural design pattern in addition to the common features and capabilities mentioned earlier.

Windows Server 2012 R2 Feature	Key Scenarios
Quality of Service (QoS) minimum bandwidth	Assigns a certain amount of bandwidth to a given type of traffic and helps make sure that each type of network traffic receives up to its assigned bandwidth.
Storage Quality of Service (QoS)	Provides storage performance isolation in a multitenant environment and mechanisms to notify you when the storage I/O performance does not meet defined thresholds.
Shared virtual hard disks	Shared virtual hard disks (.vhdx file) can be used as shared storage for multiple virtual machines that are configured as a guest failover cluster. This avoids the need to use iSCSI in this scenario.
Storage spaces	Enables cost-effective, optimally used, high availability, scalable, and flexible storage solutions in virtualized or physical deployments.
Storage Spaces tiering	Enables the creation of virtual disks that are comprised of two tiers of storage—a solid-state drive tier for frequently accessed data and a hard disk drive tier for less-frequently accessed data. Storage Spaces transparently moves data between the two tiers based on how frequently data is accessed.
Hyper-V over SMB	Supports use of SMB 3.0 file shares as storage locations for running virtual machines by using low-latency RDMA network connectivity.
SMB Direct	Provides low latency SMB 3.0 connectivity when using remote direct memory access (RDMA) adapters.
Data deduplication	Involves finding and removing duplication within data without compromising its fidelity or integrity.
SMB Multichannel	Allows file servers to use multiple network connections simultaneously, which provides increased throughput and network fault tolerance.
Virtual Receive Side Scaling (vRSS)	Allows virtual machines to support higher networking traffic loads by distributing the processing across multiple cores on the Hyper-V host and virtual machine.

Table 2 Windows Server 2012 R2 features and key scenarios

Key drivers that would encourage customers to select the Software Defined Infrastructure pattern include lower cost of ownership and flexibility with shared SAS JBOD storage solutions (Variation A only). Decision points for this design pattern over others focus primarily on the storage aspects of the solution in combination with the innovative networking capabilities of SMB Multichannel and RDMA.

4 Non-Converged Infrastructure Pattern Overview

The non-converged infrastructure pattern uses Hyper-V and Failover Clustering in a standard deployment with non-converged storage (traditional SAN) and a network infrastructure. The storage network and network paths are isolated by using dedicated I/O adapters. Failover and scalability are achieved on the storage network through Multipath I/O (MPIO). The TCP/IP network uses NIC Teaming.

In this pattern, Fibre Channel or iSCSI is expected to be the primary connectivity to a shared storage network. High-speed 10 gigabit Ethernet (GbE) adapters are common for advanced configurations of TCP/IP traffic.

Figure 6 provides an overview of the non-converged infrastructure pattern.

Figure 6 Non-converged design pattern

The non-converged infrastructure pattern has two connectivity variations:

· Variation A: Fibre Channel

· Variation B: iSCSI

Figure 7 outlines a conceptual view of this pattern.

Figure 7 Non-converged design pattern variations

Although the following list of requirements is not comprehensive, this design pattern uses the following features:

· All the common components listed earlier

· Fibre channel, iSCSI, or SMB 3.0-enabled SAN-based storage

· Storage-array support for offloaded data transfer (ODX) (optional)

Table 3 outlines the Windows Server 2012 R2 features that are utilized in this architectural design pattern in addition to the common features and capabilities outlined earlier.

Windows Server 2012 Feature	Key Scenarios
Virtual machine guest clustering enhancements (iSCSI, Virtual Fibre Channel, or shared virtual hard disks)	Supports virtual machine guest clusters by using iSCSI connections or by using the Hyper-V Fibre Channel adapter to connect to shared storage. Alternatively, a shared virtual hard disk feature can be used regardless of the shared storage protocol that is used at the host level.
Offloaded data transfer (ODX)	Support for storage-level transfers that use ODX technology (SAN feature).
Diskless network boot with iSCSI  Target Server	Provides the network-boot capability on commodity hardware by using an iSCSI boot–capable network adapter or a software boot loader (such as iPXE or netBoot/i).

Table 3 Windows Server 2012 R2 features and key scenarios

Key drivers that would encourage customers to select this design pattern include current capital and intellectual investments in SAN and transformation scenarios that include using an existing infrastructure for upgrading to a newer platform. Decision points for this design pattern include storage investments, familiarity, and flexibility of hardware.

5 Converged Infrastructure Pattern Overview

In this context, a “converged infrastructure” refers to sharing a network topology between traditional network and storage traffic. This typically implies Ethernet network devices and network controllers that have particular features to provide segregation, quality of service (performance), and scalability. The result is a network fabric that features less physical complexity, greater agility, and lower costs than those that are associated with traditional Fiber Channel-based storage networks.

This topology supports many storage designs, including traditional SANs, SMB 3.0-enabled SANs, and Windows-based Scale-Out File Server clusters. In a converged infrastructure, all storage connectivity is network-based, and it uses a single media (such as copper). SFP+ adapters are more commonly used.

Servers for a converged infrastructure pattern typically include converged blade systems and rack-mount servers, which also are prevalent in other design patterns. The key differentiators in this pattern are how the servers connect to storage and the advanced networking features that are provided by converged network adapters (CNA). High-density blade systems are common, which feature advanced hardware options that present physical or virtual network adapters to the Hyper-V host that is supporting a variety of protocols.

Figure 8 depicts a configuration for the converged infrastructure. Note the following:

· Host storage adapters can be physical or virtual, and they must support iSCSI, Fibre Channel over Ethernet (FCoE), and optionally SMB Direct.

·
Many storage devices are supported, including traditional SANs and SMB Direct–capable storage.

Figure 8 Converged infrastructure pattern

Although the following list of requirements is not comprehensive, the converged infrastructure pattern uses the following features:

· All the common components listed earlier

· Fibre channel, iSCSI, or SMB 3.02-enabled SAN-based storage

· Storage-array support for ODX (optional)

Table 4 outlines Windows Server 2012 R2 features that are utilized in the converged infrastructure pattern in addition to the common features and capabilities outlined earlier.

Windows Server 2012 Feature	Key Scenarios
Virtual machine guest clustering enhancements (iSCSI, Virtual Fibre Channel, or shared virtual hard disks)	Supports virtual machine guest clusters by using iSCSI connections or by using the Hyper-V Fibre Channel adapter to connect to shared storage. Alternatively, the shared virtual hard disk feature can be used regardless of the shared storage protocol that is used on the host level.
Offloaded data transfer (ODX)	Support for storage-level transfers that use ODX technology (SAN feature).

Table 4 Windows Server 2012 R2 features and key scenarios

6 Hybrid Infrastructure Pattern Overview

The Hybrid Infrastructure pattern includes reference architectures, best practices, and processes for extending a private cloud infrastructure to Microsoft Azure or a Microsoft service-provider partner for hybrid cloud scenarios such as:

· Extending the data center fabric to the cloud

· Extending fabric management to the cloud

· Hybrid deployment of Microsoft applications

The overall Microsoft “Cloud OS” strategy supports this architecture and approach. For more information about this strategy, see:

· Microsoft Server and Cloud Platform

· What is the Cloud OS (blog)

The key attribute of the Cloud OS vision is the hybrid infrastructure, in which you have the option to leverage an on-premises infrastructure, a Microsoft Azure infrastructure, or a Microsoft hosting partner infrastructure. Your IT organization is a consumer and a provider of services, enabling workload and application development teams to make sourcing selections for services from all three of the possible infrastructures or create solutions that span them.

The following diagram illustrates the infrastructure level, the cloud service catalog space, and examples of application scenarios and service-sourcing selections (for example, a workload team determining if it will use virtual machines that are provisioned on-premises, in Microsoft Azure, or by a Microsoft hosting partner.)

Figure 9 Hybrid IT infrastructure

By having a hybrid infrastructure in place, IT consumers focus on the service catalog instead of infrastructure. Historically, full supporting stacks would be designed from hardware, through operating system and application stack. Workloads in a hybrid environment draw from the service catalog that is provided by IT, which consists of services that are delivered by the hybrid infrastructure.

As an example, all three hybrid infrastructure pattern choices provide virtual machines; but in each case, those virtual machines have different attributes and costs. The consumer will have the choice of which one or which combination to utilize. Some virtual machines might be very low-cost but have limited features available, while others might be higher-cost but support more capability.

The hybrid infrastructure pattern enables customers to utilize private, public, and service provider clouds, each of which utilize the same product and architecture foundations.

7 Storage Architecture

7.1 Drive Architectures

The type of hard drives in the host server or in the storage array that are used by the file servers have significant impact on the overall performance of the storage architecture. The critical performance factors for hard drives are:

· The interface architecture (for example, SAS or SATA)

· The rotational speed of the drive (for example, 10K, or 15K RPM) or a solid-state drive (SSD) that does not have moving parts

· The Read and Write speed

· The average latency in milliseconds (ms)

Additional factors, such as the cache on the drive, and support for advanced features, such as Native Command Queuing (NCQ), TRIM (SATA only), Tagged Command Queuing, and UNMAP (SAS and Fibre Channel) can improve performance and duration.

As with the storage connectivity, high I/O operations per second (IOPS) and low latency are more critical than maximum sustained throughput when it comes to sizing and guest performance on the server running Hyper-V. This translates into selecting drives that have the highest rotational speed and lowest latency possible, and choosing when to use SSDs for extreme performance.

7.1.1 Serial ATA (SATA)

Serial ATA (SATA) drives are a low-cost and relatively high-performance option for storage. SATA drives are available primarily in the 3 Gbps and 6 Gbps standards (SATA II and SATA III), with a rotational speed of 7,200 RPM and average latency of around four milliseconds. Typically, SATA drives are not designed to enterprise-level standards of reliability, although new technologies in Windows Server 2012 R2 such as the Resilient File System (ReFS) can help make SATA drives a viable option for single server scenarios. However, SAS disks are required for all cluster and high availability scenarios that use Storage Spaces.

7.1.2 SAS

Serial attached SCSI (SAS) drives are typically more expensive than SATA drives, but they can provide higher performance in throughput, and more importantly, low latency. SAS drives typically have a rotational speed of 10K or 15K RPM with an average latency of 2ms to 3ms and 6 Gbps interfaces.

There are also SAS SSDs. Unlike SATA drives, SAS drives have dual interface ports that are required for using clustered storage spaces. (Details are provided in subsequent sections.) The SCSI Trade Association has a range of information about SAS drives. In addition, you can find several white papers and solutions on the LSI website.

The majority of SAN arrays today use SAS drives; but a few higher end arrays also use Fibre Channel, SAS drives, and SATA drives. For example, SAS drives are used in the Software Defined Infrastructure pattern in conjunction with a JBOD storage enclosure, which enables the Storage Spaces feature. Aside from the enclosure requirements that will be outlined later, the following requirements exist for SAS drives when they are used in this configuration:

· Drives must provide port association. Windows depends on drive enclosures to provide SES-3 capabilities such as drive-slot identification and visual drive indications (commonly implemented as drive LEDs). Windows matches a drive in an enclosure with SES-3 identification capabilities through the port address of the drive. The computer hosts can be separate from drive enclosures or integrated into drive enclosures.

· Multiport drives must provide symmetric access. Drives must provide the same performance for data-access commands and the same behavior for persistent reservation commands that arrive on different ports as they provide when those commands arrive on the same port.

· Drives must provide persistent reservations. Windows can use physical disks to form a storage pool. From the storage pool, Windows can define virtual disks, called storage spaces. A failover cluster can create high availability for a pool of physical disks, the storage spaces that they define, and the data that they contain. In addition to the standard Windows Hardware Compatibility Test qualification, physical disks should pass through the Microsoft Cluster Configuration Validation Wizard.

In addition to the drives, the following enclosure requirements exist:

· Drive enclosures must provide drive-identification services. Drive enclosures must provide numerical (for example, a drive bay number) and visual (for example, a failure LED or a drive-of-interest LED) drive-identification services. Enclosures must provide this service through SCSI Enclosure Service (SES-3) commands. Windows depends on proper behavior for the following enclosure services. Windows correlates enclosure services to drives through protocol-specific information and their vital product data page 83h inquiry association type 1.

· Drive enclosures must provide direct access to the drives that they house. Enclosures must not abstract the drives that they house (for example, form into a logical RAID disk). If they are present, integrated switches must provide discovery of and access to all of the drives in the enclosure, without requiring additional physical host connections. If possible, multiple host connections must provide discovery of and access to the same set of drives.

Hardware vendors should pay specific attention to these storage drive and enclosure requirements for SAS configurations when they are used in conjunction with the Storage Spaces feature in Windows Server 2012 R2.

7.1.3 Nearline SAS (NL-SAS)

Nearline SAS (NL-SAS) drives deliver the larger capacity benefits of enterprise SATA drives with a fully capable SAS interface. The physical properties of the drive are identical to those of traditional SATA drives, with a rotational speed of 7,200 RPM and average latency of around four milliseconds. However, exposing the SATA drive through a SAS interface provides all the enterprise features that come with an SAS drive, including multiple host support, concurrent data channels, redundant paths to the disk array (required for clustered storage spaces), and enterprise command queuing. The result is SAS drives that are capable of much larger capacities and available at significantly lower costs.

It is important to consider that although NL-SAS drives provide greater capacity through the SAS interface, they have the same latency, reliability, and performance limitations of traditional enterprise SATA drives, which results in the higher drive failure rates of SATA drives compared to native SAS drives.

As a result, NL-SAS drives can be used in cluster and high availability scenarios that use Storage Spaces, although using SSD drives to implement storage tiers is highly recommended to improve storage performance.

7.1.4 Fibre Channel

Fibre Channel is traditionally used in SAN arrays and it provides high speed (same as SAS), low latency, and enterprise-level reliability. Fibre Channel is usually more expensive than using SATA or SAS drives. Fibre Channel typically has performance characteristics that are similar to those of SAS drives, but it uses a different interface. The choice of Fibre Channel or SAS drives is usually determined by the choice of storage array or drive tray. In many cases, SSDs can also be used in SAN arrays that use Fibre Channel interfaces. All arrays also support SATA devices, sometimes with an adapter. Disk and array vendors have largely transitioned to SAS drives.

7.1.5 Solid-State Storage

SATA, SAS, NL-SAS, and Fibre Channel describe the interface type, and a solid-state drive (SSD) refers to the classification of media type. Solid-state storage has several advantages over traditional spinning media disks, but it comes at a premium cost. The most prevalent type of solid-state storage is a solid-state drive. Some advantages include significantly lower latency, no spin-up time, faster transfer rates, lower power and cooling requirements, and no fragmentation concerns.

Recent years have shown greater adoption of SSDs in enterprise storage markets. These more expensive devices are usually reserved for workloads that have high-performance requirements. Mixing SSDs with spinning disks in storage arrays is common to minimize cost. These storage arrays often have software algorithms that automatically place the frequently accessed storage blocks on the SSDs and the less frequently accessed blocks on the lower-cost disks (referred to as auto-tiering), although manual segregation of disk pools is also acceptable. NAND Flash Memory is most commonly used in SSDs for enterprise storage.

7.1.6 Hybrid Drives

Hybrid drives combine traditional spinning disks with nonvolatile memory or small SSDs that act as a large buffer. This method provides the potential benefits of solid-state storage with the cost effectiveness of traditional disks. Currently, these drives are not commonly found in enterprise storage arrays.

7.1.7 Advanced Format (4K) Disk Compatibility

Windows Server 2012 introduced support for large sector disks that support 4096-byte sectors (referred to as 4K), rather than the traditional 512-byte sectors, which ship with most hard drives. This change offers higher capacity drives, better error correction, and more efficient signal-to-noise ratios. Windows Server 2012 R2 provides continued support for this format and these drives are becoming more prevalent in the market.

However, this change introduces compatibility challenges. To support compatibility, two types of 4K drives exist: 512-byte emulation (512e) and 4K native. 512e drives present a 512-byte logical sector to use as the unit of addressing, and they present a 4K physical sector to use as the unit of atomic write (the unit defined by the completion of Read and Write operations in a single operation).

7.2 Storage Controller Architectures

For servers that will be directly connected to storage devices or arrays (which could be servers running Hyper-V or file servers that will provide storage), the choice of storage controller architecture is critical to performance, scale, and overall cost.

7.2.1 SATA III

SATA III controllers can operate at speeds of up to 6 Gbps, and enterprise-oriented controllers can include varying amounts of cache on the controller to improve performance.

SAS controllers can also operate at speeds of up to 6 Gbps, and they are more common in server form factors than SATA. If you are running Windows Server 2012 R2, it is important to understand the difference between host bus adapters (HBAs) and RAID controllers.

SAS HBAs provide direct access to the disks, trays, or arrays that are attached to the controller. There is no controller-based RAID. Disk high availability is provided by the array or by the tray. In Storage Spaces, high availability is provided by higher-level software layers. SAS HBAs are common in one-, two-, and four-port models.

To support the Storage Spaces feature in Windows Server 2012 R2, the HBA must report the HBA that is used to connected devices. For example, drives that are connected through the SAS HBA provide a valid configuration, whereas drives that are connected through the RAID HBA provides an invalid configuration.

All commands must be passed directly to the underlying physical devices. The physical devices must not be abstracted (that is, formed into a logical RAID device), and the HBA must not respond to commands on behalf of the physical devices.

Figure 10 Example SAS JBOD Storage architecture

7.2.2 PCIe/SAS HBA

SAS controllers can also operate at speeds of up to 6 Gbps, and they are more common in server configurations than SATA. With Windows Server 2012 R2, it is important to understand the difference between host bus adapters (HBAs) and RAID controllers. A SAS HBA provides direct access to the disks, trays, or arrays attached to the controller. There is no support for configurations which use controller-based RAID. Disk high availability is provided by either the array or tray itself, or—in the case of Storage Spaces—by higher-level software layers. SAS HBAs are common in one-, two-, and four-port models.

To support Storage Spaces, HBAs must report the physical bus that is used to connected devices (for example, drives connected via the SAS bus is a valid configuration whereas drives as connected via the RAID bus is an invalid configuration). All commands must be passed directly to the underlying physical devices. The physical devices must not be abstracted (that is, formed into a logical RAID device), and the bus adapter must not respond to commands on behalf of the physical devices.

7.2.3 PCIe RAID/Clustered RAID

Peripheral Component Interconnect Express (PCIe) RAID controllers are the traditional cards that are found in servers. They provide access to storage systems, and they can include RAID technology. RAID controllers are not typically used in cluster scenarios, because clustering requires shared storage. If Storage Spaces is used, RAID should not be enabled because Storage Spaces handles data availability and redundancy.

Clustered RAID controllers are a type of storage interface card that can be used with shared storage and cluster scenarios. The clustered RAID controllers can provide shared storage across configured servers. The clustered RAID controller solution must pass the Cluster in a Box Validation Kit. This step is required to make sure that the solution provides the storage capabilities that are necessary for failover cluster environments.

Figure 11 Clustered RAID controllers

7.2.4 Fibre Channel HBA

Fibre Channel HBAs provide one of the more common connectivity methods for storage, particularly in clustering and shared storage scenarios. Some HBAs include two or four ports, with ranges of four, eight, or 16 (Gen5) Gbps. Windows Server 2012 R2 supports a large number of logical unit numbers (LUNs) per HBA[1]. The capacity is expected to exceed the needs of customers for addressable LUNs in a SAN.

Hyper-V in Windows Server 2012 R2 provides the ability to support virtual Fibre Channel adapters within a guest operating system. This is not necessarily required even if Fibre Channel presents storage to servers running Hyper-V

Although virtual Fibre Channel is discussed in later sections of this document, it is important to understand that the HBA ports that are used with virtual Fibre Channel should be set up in a Fibre Channel topology that supports N_Port ID Virtualization (NPIV), and they should be connected to an NPIV-enabled SAN. To utilize this feature, the Fibre Channel adapters must also support devices that present LUNs.

7.3 Storage Networking

A variety of storage networking protocols exist to support traditional SAN-based scenarios, network-attached storage scenarios, and the newer software-defined infrastructure scenarios that support file-based storage through the use of SMB.

7.3.1 Fibre Channel

Historically, Fibre Channel has been the storage protocol of choice for enterprise data centers for a variety of reasons, including performance and low latency. These considerations have offset the typically higher costs of Fibre Channel. The continually advancing performance of Ethernet from one Gbps to 10 Gbps and beyond has led to great interest in storage protocols that use Ethernet transports, such as iSCSI and Fibre Channel over Ethernet (FCoE).

Given the long history of Fibre Channel in the data center, many organizations have a significant investment in a Fibre Channel–based SAN infrastructure. Windows Server 2012 R2 continues to provide full support for Fibre Channel hardware that is logo-certified. There is also support for virtual Fibre Channel in guest virtual machines through a Hyper-V feature in Windows Server 2012 and Windows Server 2012 R2.

7.3.2 iSCSI

In Windows Server 2012 R2 and Windows Server 2012, the iSCSI Target Server is available as a built-in option under the file and storage service role instead of a separate downloadable add-on, so it is easier to deploy. The iSCSI Target Server capabilities were enhanced to support diskless network-boot capabilities. This demonstrates how the storage protocols in Windows Server 2012 R2 and Windows Server 2012 are designed to complement each other across all layers of the storage stack.

In Windows Server 2012 R2 and Windows Server 2012, the iSCSI Target Server feature provides a network-boot capability from operating system images that are stored in a centralized location. This capability supports commodity hardware for up to 256 computers.

Windows Server 2012 R2 supports up to a maximum of 276 LUNs and a maximum number of 544 sessions per target. In addition, this capability does not require special hardware, but it is recommended to be used in conjunction with 10 GbE adapters that support iSCSI boot capabilities. For Hyper-V, iSCSI-capable storage provides an advantage because it is the protocol that is utilized by Hyper-V virtual machines for clustering.

Initially introduced in Hyper-V in Windows Server 2012, the iSCSI Target Server uses the VHDX format to provide a storage format for LUNs. The VHDX format provides data corruption protection during power failures and optimizes structural alignments of dynamic and differencing disks to prevent performance degradation on large-sector physical disks. This also provides the ability to provision target LUNs up to 64 TB and the ability to provision fixed-size and dynamically growing disks. In Windows Server 2012 R2, all new disks that are created in the iSCSI Target Server use the VHDX format; however, standard disks with the VHD format can be imported.

In addition, the iSCSI Target Server in Windows Server 2012 R2 enables Force Unit Access (FUA) for I/O on its back-end virtual disk only if the front-end that the iSCSI Target Server received from the initiator requires it. This has the potential to improve performance, assuming FUA-capable back-end disks or JBODs are used with the iSCSI Target Server.

7.3.3 Fibre Channel over Ethernet (FCoE)

A key advantage of the protocols using an Ethernet transport is the ability to use a converged network architecture. Converged networks have an Ethernet infrastructure that serves as the transport for LAN and storage traffic. This can reduce costs by eliminating dedicated Fibre Channel switches and reducing cables.

Fibre Channel over Ethernet (FCoE) allows the potential benefits of using an Ethernet transport, while retaining the advantages of the Fibre Channel protocol and the ability to use Fibre Channel storage arrays.

Several enhancements to standard Ethernet are required for FCoE. The enriched Ethernet is commonly referred to as enhanced Ethernet or Data Center Ethernet. These enhancements require Ethernet switches that are capable of supporting enhanced Ethernet.

7.3.4 InfiniBand

InfiniBand is an industry-standard specification that defines an input/output architecture that is used to interconnect servers, communications infrastructure equipment, storage, and embedded systems. InfiniBand is a true fabric architecture that utilizes switched, point-to-point channels with data transfers of up to 120 gigabits per second (Gbps), in chassis backplane applications and through external copper and optical fiber connections.

InfiniBand provides a low-latency, high-bandwidth interconnection that requires low processing overhead. It is ideal for carrying multiple traffic types (such as clustering, communications, storage, and management) over a single connection.

7.3.5 Switched SAS

Although switched SAS is not traditionally viewed as a storage networking technology, it is possible to design switched SAS storage infrastructures. In fact, this can be a low cost and powerful approach when combined with Windows Server 2012 R2 features, such as Storage Spaces and SMB 3.0.

SAS switches enable multiple host servers to be connected to multiple storage trays (SAS JBODs) with multiple paths between each, as shown in Figure 10. Multiple path SAS implementations use a single domain method to provide fault tolerance. Current mainstream SAS hardware supports 6 Gbps. SAS switches support domains that enable functionality similar to zoning in Fibre Channel.

Figure 12 SAS switch connected to multiple SAS JBOD arrays

7.3.6 Network File System

File-based storage is a practical alternative to more SAN storage because it is straightforward to provision. It has gained viability because it is simple to provision and manage. An example of this trend is the popularity of deploying and running VMware vSphere virtual machines from file-based storage that is accessed over the Network File System (NFS) protocol.

To help you utilize this, Windows Server 2012 R2 includes an updated Server for NFS that supports NFS 4.1 and can utilize many other performance, reliability, and availability enhancements that are available throughout the storage stack in Windows. Some of the key features that are available in Server for NFS include:

· Storage for VMware virtual machines over NFS. In Windows Server 2012 R2, you can confidently deploy Server for NFS as a high availability storage back end for VMware virtual machines. Critical components of the NFS stack have been designed to provide transparent failover semantics to NFS clients.

· NFS 4.1 protocol. The NFS 4.1 protocol is a significant evolution, and Microsoft delivers a standards-compliant server-side implementation in Windows Server 2012 R2. Some of the features of NFS 4.1 include a flexible single-server namespace for easier share management, full Kerberos version 5 support for enhanced security (including authentication, integrity, and privacy), VSS snapshot integration for backup, and Unmapped UNIX User Access for easier user account integration. Windows Server 2012 R2 supports simultaneous SMB 3.0 and NFS access to the same share, identity mapping by using stores based on RFC-2307 for easier and more secure identity integration, and high availability cluster deployments.

· Windows PowerShell. In response to customer feedback, over 40 Windows PowerShell cmdlets provide task-based remote management for every aspect of Server for NFS—from the configuring NFS server settings to provisioning shares and share permissions.

· Simplified identity mapping. Windows Server 2012 R2 includes a flat file–based identity-mapping store. Windows PowerShell cmdlets replace cumbersome manual steps to provision Active Directory Lightweight Directory Services (AD LDS) as an identity-mapping store and to manage mapped identities.

7.3.7 SMB 3.0

Similar to the other file-based storage options that were discussed earlier, Hyper-V and SQL Server can take advantage of Server Message Block (SMB)-based storage for virtual machines and SQL Server database data and logs. This support is enabled by the features and capabilities provided in Windows Server 2012 R2 and Windows Server 2012 as outlined in the following sections.

Note  It is strongly recommended that Scale-Out File Server is used in conjunction with IaaS architectures that use SMB storage as discussed in the following sections.

Figure 13 Example of a SMB 3.0-enabled network-attached storage

7.3.7.1 SMB Direct (SMB over RDMA)

The SMB protocol in Windows Server 2012 R2 includes support for remote direct memory access (RDMA) on network adapters, which allows storage-performance capabilities that rival Fibre Channel. RDMA enables this performance capability because network adapters can operate at full speed with very low latency due to their ability to bypass the kernel and perform Read and Write operations directly to and from memory. This capability is possible because effective transport protocols are implemented on the network adapter hardware, which allows for zero-copy networking by bypassing the kernel.

By using this capability, applications (including SMB) can transfer data directly from memory, through the network adapter, to the network, and then to the memory of the application that is requesting data from the file share. This means that two kernel calls (one from the server and one from the client) are largely removed from the data transfer process, which results in greatly improved data transfer performance. This capability is especially useful for Read and Write intensive workloads, such as in Hyper-V or Microsoft SQL Server, and it results in remote file server performance that is comparable to local storage.

SMB Direct requires:

· At least two computers running Windows Server 2012 R2. No additional features have to be installed, and the technology is available by default.

· Network adapters that are RDMA-capable with the latest vendor drivers installed. SMB Direct supports common RDMA-capable network adapter types, including Internet Wide Area RDMA Protocol (iWARP), InfiniBand, and RDMA over Converged Ethernet (RoCE).

SMB Direct works in conjunction with SMB Multichannel to transparently provide a combination of exceptional performance and failover resiliency when multiple RDMA links between clients and SMB file servers are detected. Be aware that RDMA bypasses the kernel stack, and therefore RDMA does not work with NIC Teaming; however, it does work with SMB Multichannel, because SMB Multichannel is enabled at the application layer. The result being that SMB multichannel provides the equivalent of NIC teaming for access to SMB file shares.

In Windows Server 2012 R2, SMB Direct optimizes I/O with high-speed network adapters, including 40 Gbps Ethernet and 56 Gbps InfiniBand through the use of batching operations, RDMA remote invalidation, and non-uniform memory access (NUMA) optimizations. In Windows Server 2012 R2, SMB Direct also leverages Network Direct Kernel Provider Interface 1.2 (NDKPI 1.2), and it is backwards compatible with NDKPI 1.1.

7.3.7.2 SMB Multichannel

SMB 3.0 protocol in Windows Server 2012 R2 supports SMB Multichannel, which provides scalable and resilient connections to SMB shares that dynamically create multiple connections for single sessions or multiple sessions on single connections, depending on connection capabilities and current demand. This capability to create flexible session-to-connection associations gives SMB a number of key features, including:

· Connection resiliency: With the ability to dynamically associate multiple connections with a single session, SMB gains resiliency against connection failures that are usually caused by network interfaces or components. SMB Multichannel also allows clients to actively manage paths of similar network capability in a failover configuration that automatically switches sessions to the available paths if one path becomes unresponsive.

· Network usage: SMB can utilize receive-side scaling (RSS)–capable network interfaces with the multiple connection capability of SMB Multichannel to fully use high-bandwidth connections, such as those that are available on 10 GbE networks, during Read and Write operations with workloads that are evenly distributed across multiple CPUs.

· Load balancing: Clients can adapt to changing network conditions to dynamically rebalance loads to a connection or across a set of connections that are more responsive when congestion or other performance issues occur.

· Transport flexibility: Because SMB Multichannel also supports single session to multiple connection capabilities, SMB clients are flexible enough to adjust dynamically when new network interfaces become active. This is how SMB Multichannel is automatically enabled whenever multiple UNC paths are detected and can grow dynamically to use multiple paths as more are added, without administrator intervention.

SMB Multichannel has the following requirements, which are organized by how SMB Multichannel prioritizes connections when multiple connection types are available:

· RDMA-capable network connections: SMB Multichannel can be used with a single InfiniBand connection on the client and server or with a dual InfiniBand connection on each server, connected to different subnets. Although SMB Multichannel offers scaling performance enhancements in single adapter scenarios through RDMA and RSS, if available, it cannot supply failover and load balancing capabilities without multiple paths. RDMA-capable network adapters include iWARP, InfiniBand, and RoCE.

· RSS-capable network connections: SMB Multichannel can utilize RSS-capable connections in 1-1 connection scenarios or multiple connection scenarios. Multichannel load balancing and failover capabilities are not available unless multiple paths exist, but it provides scaling performance usage by spreading overhead between multiple processors by using RSS-capable hardware.

· Load balancing and failover or aggregate interfaces: When RDMA or RSS connections are not available, SMB prioritizes connections that use a collection of two or more physical interfaces. This requires more than one network interface on the client and server, where both are configured as a network adapter team. In this scenario, load balancing and failover are the responsibility of the teaming protocol, not SMB Multichannel, when only one NIC Teaming connection is present and no other connection path is available.

· Standard interfaces and Hyper-V virtual networks: These connection types can use SMB Multichannel capabilities but only when multiple paths exist. For all practical intentions, one GB Ethernet connection is the lowest priority connection type that is capable of using SMB Multichannel.

· Wireless network interfaces: Wireless interfaces are not capable of multichannel operations.

When connections are not similar between client and server, SMB Multichannel utilizes available connections when multiple connection paths exist. For example, if the SMB file server has a 10 GbE connection, but the client has only four 1 GbE connections, and each connection forms a path to the file server, then SMB Multichannel can create connections on each 1 GbE interface. This provides better performance and resiliency, even though the network capabilities of the server exceed the network capabilities of the client.

Note that SMB Multichannel only affects multiple file operations and you cannot distribute a single file operation (such as accessing a particular VHD) over multiple channels simultaneously. However, a single file copy or accessing a single file on a VHD uses multiple channels although each Read or Write operation travels through only one of the channels.

7.3.7.3 SMB Transparent Failover

SMB Transparent Failover helps administrators configure file shares in Windows failover cluster configurations for continuous availability. Continuous availability enables administrators to perform hardware or software maintenance on any cluster node without interrupting the server applications that are storing their data files on these file shares.

If there is a hardware or software failure, the server application nodes transparently reconnect to another cluster node without interrupting the server application I/O operations. By using Scale-Out File Server, SMB Transparent Failover allows the administrator to redirect a server application node to a different file-server cluster node to facilitate better load balancing.

SMB Transparent Failover has the following requirements:

· A failover cluster that is running Windows Server 2012 R2 with at least two nodes. The configuration of servers, storage, and networking must pass all of the tests performed in the Cluster Configuration Validation Wizard.

· Scale-Out File Server role installed on all cluster nodes.

· Clustered file server configured with one or more file shares that have continuous availability.

· Client computers running Windows Server 2012 R2, Windows Server 2012, Windows 8.1, or Windows 8.

To realize the potential benefits of the SMB Transparent Failover feature, the client computer and the server must support SMB 3.0, which was first introduced in Windows Server 2012 and Windows 8. Computers running down-level SMB versions, such as SMB 2.1 or SMB 2.0 can connect and access data on a file share that has continuous availability, but they will not be able to realize the potential benefits of the SMB Transparent Failover feature. It is important to note that Hyper-V over SMB requires SMB 3.0; therefore, down-level versions of the SMB protocol are not relevant for these designs.

7.3.7.4 SMB Encryption

SMB Encryption protects incoming data from unintentional snooping threats on untrusted networks, with no additional setup requirements. SMB 3.0 in Windows Server 2012 R2 secures data transfers by encrypting incoming data, to protect against tampering and eavesdropping attacks. The biggest potential benefit of using SMB Encryption instead of general solutions (such as IPsec) is that there are no deployment requirements or costs beyond changing the SMB settings in the server. The encryption algorithm that is used is AES-CCM, which also provides data-integrity validation.

SMB 3.0 uses a newer algorithm (AES-CMAC) for validation, instead of the HMAC-SHA-256 algorithm that SMB 2.0 uses. AES-CCM and AES-CMAC can be dramatically accelerated on most modern CPUs that have AES instruction support.

By using Windows Server 2012 R2, an administrator can enable SMB Encryption for the entire server, or for only specific file shares. Because there are no other deployment requirements for SMB Encryption, it is an extremely cost effective way to protect data from snooping and tampering attacks. Administrators can turn on SMB Encryption simply by using the File Server Manager or Windows PowerShell.

7.3.7.5 Volume Shadow Copy Service

Volume Shadow Copy Service (VSS) is a framework that enables volume backups to run while applications on a system continue to write to the volumes. A new feature called “VSS for SMB File Shares” was introduced in Windows Server 2012 R2 to support applications that store their data files on remote SMB file shares. This feature enables VSS-aware backup applications to perform consistent shadow copies of VSS-aware server applications that store data on SMB 3.0 file shares. Prior to this feature, VSS supported only shadow copies of data that were stored on local volumes.

7.3.7.6 Scale-Out File Server

One the main advantages of file storage over block storage is the ease of configuration, paired with the ability to configure folders that can be shared by multiple clients. SMB takes this one step farther by introducing the SMB Scale-Out feature, which provides the ability to share the same folders from multiple nodes of the same cluster. This is made possible by the use of the Cluster Shared Volumes (CSV) feature, which supports file sharing in Windows Server 2012 R2.

For example, if you have a four-node file-server cluster that uses Scale-Out File Server, an SMB client will be able to access the share from any of the four nodes. This active-active configuration lets you balance the load across cluster nodes by allowing an administrator to move clients without any service interruption. This means that the maximum file-serving capacity for a given share is no longer limited by the capacity of a single cluster node.

Scale-Out File Server also helps keep configurations simple, because a share is configured only once to be consistently available from all nodes of the cluster. Additionally, SMB Scale-Out simplifies administration by not requiring cluster virtual IP addresses or by creating multiple cluster file-server resources to utilize all cluster nodes.

Scale-Out File Server requires:

· A failover cluster that is running Windows Server 2012 R2 with at least two nodes. The cluster must pass the tests in the Cluster Configuration Validation Wizard. In addition, the clustered role should be created for scale out. This is not applicable to traditional file server clustered roles. An existing traditional file server clustered role cannot be used in (or upgraded to) a Scale-Out File Server cluster.

· File shares that are created on a Cluster Shared Volume with continuous availability. This is the default setting.

· Computers running Windows Server 2012 R2, Windows Server 2012, Windows 8.1, or Windows 8.

Windows Server 2012 R2 provides several capabilities with respect to Scale-Out File Server functionality including support for multiple instances, bandwidth management, and automatic rebalancing. SMB in Windows Server 2012 R2 provides an additional instance on each cluster node in Scale-Out File Server specifically for CSV traffic. A default instance can handle incoming traffic from SMB clients that are accessing regular file shares, while another instance only handles inter-node CSV traffic.

The SMB in Windows Server 2012 R2 uses data structures (locks, queues, and threads) to satisfy requests between clients and cluster nodes. In Windows Server 2012 R2, each node contains two logical instances of SMB—one instance to handle CSV metadata or redirected traffic between nodes and a second instance to handle SMB clients that are accessing file share data. Windows Server 2012 R2 provides separate data structures (locks and queues) for each type of traffic, improving the scalability and reliability of traffic between nodes.

SMB in Windows Server 2012 R2 also supports the ability to configure bandwidth limits based for predefined categories. SMB traffic is divided into three predefined categories that are named Default, VirtualMachine, and LiveMigration. It is possible to configure a bandwidth limit for each predefined category through Windows PowerShell or WMI by using bytes-per-second. This is especially useful when live migration and SMB Direct (SMB over RDMA) are utilized.

In Windows Server 2012 R2, Scale-Out File Server also supports automatic rebalancing. SMB client connections are tracked per file share (instead of per server) when direct I/O is not available on the volume. Clients are then redirected to the cluster node with the best access to the volume that is used by the file share. This automatic behavior improves efficiency by reducing redirection traffic between file server nodes. Clients are redirected following an initial connection and when cluster storage is reconfigured.

7.4 File Services

7.4.1 Storage Spaces

Storage Spaces introduces a new class of sophisticated storage virtualization enhancements to the storage stack that incorporates two concepts:

· Storage pools: Virtualized units of physical disk units that enable storage aggregation, elastic capacity expansion, and delegated administration.

· Storage spaces: Virtual disks with associated attributes that include a desired level of resiliency, thin or fixed provisioning, automatic or controlled allocation on diverse storage media, and precise administrative control.

The Storage Spaces feature in Windows Server 2012 R2 can utilize failover clustering for high availability, and it can be integrated with CSV for scalable deployments.

Figure 14 CSV v2 can be integrated with Storage Spaces

The features that Storage Spaces includes are:

· Storage pooling: Storage pools are the fundamental building blocks for Storage Spaces. IT administrators can flexibly create storage pools, based on the needs of the deployment. For example, given a set of physical disks, an administrator can create one pool by using all of the physical disks that are available or multiple pools by dividing the physical disks as required. In addition, to promote the value from storage hardware, the administrator can map a storage pool to combinations of hard disk drives in addition to solid-state drives (SSDs). Pools can be expanded dynamically simply by adding more drives, thereby seamlessly scaling to cope with increasing data growth as needed.

· Multitenancy: Administration of storage pools can be controlled through access control lists (ACLs) and delegated on a per-pool basis, thereby supporting hosting scenarios that require tenant isolation. Storage Spaces follows the familiar Windows security model. Therefore, it can be integrated fully with Active Directory Domain Services (AD DS).

· Resilient storage: Storage Spaces supports two optional resiliency modes: mirroring and parity. Capabilities such as per-pool hot spare support, background scrubbing, and intelligent error correction enable optimal service availability despite storage component failures.

· Continuous availability through integration with failover clustering: Storage Spaces is fully integrated with failover clustering to deliver continuous availability. One or more pools can be clustered across multiple nodes in a single cluster. Storage Spaces can then be instantiated on individual nodes, and it will seamlessly migrate or fail over to a different node in response to failure conditions or because of load balancing. Integration with CSV 2.0 enables scalable access to data on storage infrastructures.

· Optimal storage use: Server consolidation frequently results in multiple datasets that share the same storage hardware. Storage Spaces supports thin provisioning to enable businesses to easily share storage capacity among multiple unrelated datasets, thereby promoting capacity use. Trim support enables capacity reclamation when possible.

· Operational simplicity: Fully scriptable remote management is permitted through the Windows Storage Management API, Windows Management Instrumentation (WMI), and Windows PowerShell. Storage Spaces can be managed easily through the File Services GUI in Server Manager or by using task automation with many new Windows PowerShell cmdlets.

· Fast rebuild: If a physical disk fails, Storage Spaces will regenerate the data from the failed physical disk in parallel. During parallel regeneration, a single disk in the pool serves as the source of data or the target of data, and Storage Spaces maximizes peak sequential throughput. No user action is necessary, and the newly created Storage Spaces will use the new policy.

For single-node environments, Windows Server 2012 R2 requires the following:

· Serial or SAS-connected disks (in an optional JBOD enclosure)

For multiserver and multisite environments, Windows Server 2012 R2 requires the following:

· Any requirements that are specified for failover clustering and CSV 2.0

· Three or more SAS-connected disks (JBODs) to encourage compliance with Windows Certification requirements

7.4.1.1 Storage Spaces Write-Back Cache

Storage Spaces in Windows Server 2012 R2 supports an optional write-back cache that can be configured with simple, parity, and mirrored spaces. The write-back cache is designed to improve performance for workloads with small, random writes by using solid-state drives (SSDs) to provide a low-latency cache. The write-back cache has the same resiliency requirements as the Storage Spaces it is configured for. That is, a simple space requires a single journal drive, a two-way mirror requires two journal drives, and a three-way mirror requires three journal drives.

If you have dedicated journal drives, they are automatically selected for the write-back cache, and if there are no dedicated journal drives, drives that report the media type as SSDs are selected to host the write-back cache.

The write-back cache is associated with an individual space, and it is not shared. However, the physical SSDs that are associated with the write-back cache can be shared among multiple write-back caches. The write-back cache is enabled by default, if there are SSDs available in the pool. The default size of the write-back cache is 1 GB, and it is not recommended to change the size. After the write-back cache is created, its size cannot be changed.

7.4.1.2 Storage Tiers

Storage Spaces in Windows Server 2012 R2 supports the capability for a virtual hard disk to have the best characteristics of SSDs and hard disk drives to optimize placement of workload data. The most frequently accessed data is prioritized to be placed on high-performance SSDs, and the less frequently accessed data is prioritized to be placed on high-capacity, lower-performance hard disk drives.

Data activity is measured in the background and periodically moved to the appropriate location with minimal performance impact to a running workload. Administrators can further override automated placement of files, based on access frequency. It is important to note that storage tiers are compatible only with mirror spaces or simple spaces. Parity spaces are not compatible with storage tiers.

7.4.2 Resilient File System

Windows Server 2012 R2 supports the updated local file system called Resilient File System (ReFS). ReFS promotes data availability and online operation, despite errors that would historically cause data loss or downtime. Data integrity helps protect business-critical data from errors and helps make sure that the data is available when needed. The ReFS architecture provides scalability and performance in an era of constantly growing dataset sizes and dynamic workloads.

ReFS was designed with three key goals in mind:

· Maintain the highest possible levels of system availability and reliability, under the assumption that the underlying storage might be unreliable.

· Provide a full end-to-end resilient architecture when it is used in conjunction with Storage Spaces, so that these two features magnify the capabilities and reliability of one another when they are used together.

· Maintain compatibility with widely adopted and successful NTFS file system features, while replacing features that provide limited value.

In Windows Server 2012 R2, Cluster Shared Volumes includes compatibility for ReFS in addition to many other storage enhancements. ReFS in Windows Server 2012 R2 also provides support to automatically correct corruption in a Storage Spaces parity space.

7.4.3 NTFS Improvements

In Windows Server 2012 R2 and Windows Server 2012, NTFS has been enhanced to maintain data integrity when using cost-effective, industry-standard SATA drives. NTFS provides online corruption scanning and repair capabilities that reduce the need to take volumes offline. When they are combined, these capabilities let you deploy very large NTFS volumes with confidence.

Two key enhancements were made to NTFS in Windows Server 2012. The first one targets the need to maintain data integrity in inexpensive commodity storage. This was accomplished by enhancing NTFS so that it relies on the flush command instead of “forced unit access” for all operations that require Write ordering. This improves resiliency against metadata inconsistencies that are caused by unexpected power loss. This means that you can more safely use cost-effective, industry-standard SATA drives.

NTFS availability is the focus of the second key enhancement, and this is achieved through a combination of features, which include:

· Online corruption scanning: Windows Server 2012 R2 performs online corruption scanning operations as a background operation on NTFS volumes. This scanning operation identifies and corrects areas of data corruption if they occur, and it includes logic that distinguishes between transient conditions and actual data corruption, which reduces the need for CHKDSK operations.

· Improved self-healing: To further improve resiliency and availability, Windows Server 2012 R2 significantly increases online self-healing to resolve many issues on NTFS volumes without the need to take the volume offline to run CHKDSK.

· Reduced repair times: In the rare case of data corruption that cannot be fixed with online self-healing, administrators are notified that data corruption has occurred, and they can choose when to take the volume offline for a CHKDSK operation. Furthermore, because of the online corruption-scanning capability, CHKDSK scans and repairs only tagged areas of data corruption. Because it does not have to scan the whole volume, the time that is necessary to perform an offline repair is greatly reduced. In most cases, repairs that would have taken hours on volumes that contain a large number of files, now take seconds.

7.4.4 Scale-Out File Server Cluster Architecture

In Windows Server 2012 R2, the following clustered file-server types are available:

· Scale-Out File Server cluster for application data: This clustered file server lets you store server application data (such as virtual machine files in Hyper-V) on file shares, and obtain a similar level of reliability, availability, manageability, and performance that you would expect from a storage area network. All file shares are online on all nodes simultaneously. File shares that are associated with this type of clustered file server are called scale-out file shares. This is sometimes referred to as an Active/Active cluster.

· File Server for general use: This is the continuation of the clustered file server that has been supported in Windows Server since the introduction of failover clustering. This type of clustered file server, and thus all of the file shares that are associated with the clustered file server, is online on one node at a time. This is sometimes referred to as an Active/Passive or a Dual/Active cluster. File shares that are associated with this type of clustered file server are called clustered file shares.

In Windows Server 2012 R2, a Scale-Out File Server cluster is designed to provide file shares that are continuously available for file-based server application storage. As discussed previously, scale-out file shares provide the ability to share the same folder from multiple nodes of the same cluster. For instance, if you have a four-node file-server cluster that is using the Server SMB Scale-Out feature, which was introduced in Windows Server 2012, a computer that is running Windows Server 2012 R2 or Windows Server 2012 can access file shares from any of the four nodes. This is achieved by utilizing failover-clustering features in Windows Server 2012 R2 and Windows Server 2012 and new capabilities in SMB 3.0.

File server administrators can provide scale-out file shares and continuously available file services to server applications and respond to increased demands quickly by bringing more servers online. All of this is completely transparent to the server application. When combined with Scale-Out File Server, this provides comparable capabilities to traditional SAN architectures.

Key potential benefits that are provided by a Scale-Out File Server cluster in Windows Server 2012 R2 include:

· Active/Active file shares. All cluster nodes can accept and serve SMB client requests. By making the file-share content accessible through all cluster nodes simultaneously, SMB 3.0 clusters and clients cooperate to provide transparent failover (continuous availability) to alternative cluster nodes during planned maintenance and unplanned failures without service interruption.

· Increased bandwidth. The maximum file share bandwidth is the total bandwidth of all file-server cluster nodes. Unlike in previous versions of Windows Server, the total bandwidth is not constrained to the bandwidth of a single cluster node, but instead to the capability of the backing storage system. You can increase the total bandwidth by adding nodes.

· CHKDSK with zero downtime. CHKDSK in Windows Server 2012 R2 is significantly enhanced to dramatically shorten the time a file system is offline for repair. Cluster Shared Volumes (CSV) in Windows Server 2012 take this one step further and eliminate the offline phase. A CSV file system can perform CHKDSK without affecting applications that have open handles in the file system.

· Clustered Shared Volumes cache. CSV in Windows Server 2012 R2 support a Read cache, which can significantly improve performance in certain scenarios, such as a Virtual Desktop Infrastructure.

· Simplified management. You create the Scale-Out File Server cluster and then add the necessary CSV and file shares. It is no longer necessary to create multiple clustered file servers, each with separate cluster disks, and then develop placement policies to confirm activity on each cluster node.

7.5 Storage Features

7.5.1 Data Deduplication

The Fibre Channel SAN component in iSCSI often provides data deduplication functionality. By using the data deduplication feature in Windows Server 2012 R2, organizations can significantly improve the efficiency of storage capacity usage. In Windows Server 2012 R2, data deduplication provides the following features:

· Capacity optimization: Data deduplication lets you store more data in less physical space. You can achieve significantly better storage efficiency than was previously possible with Single Instance Storage (SIS) or New Technology File System (NTFS) compression. Data deduplication uses variable size chunking and compression, which deliver optimization ratios of up to 2:1 for general file servers and up to 20:1 for VHD libraries.

· Scalability and performance: Data deduplication is highly scalable, resource-efficient, and non-intrusive. It can run on dozens of large volumes of primary data simultaneously, without affecting other workloads on the server.

· Reliability and data integrity: When you apply data deduplication, you must maintain data integrity. To help with data integrity, Windows Server 2012 R2 utilizes checksum, consistency, and identity validation. In addition, to recover data in the event of corruption, Windows Server 2012 R2 maintains redundancy for all metadata and the most frequently referenced data.

· Bandwidth efficiency with BranchCache: Through integration with BranchCache, the same optimization techniques that are applied for improving data storage efficiency on the disk are applied to transferring data over a wide area network (WAN) to a branch office. This integration results in faster file download times and reduced bandwidth consumption.

Windows Server data deduplication uses a post-processing approach, which identifies files for optimization and then applies the algorithm for deduplication. The deduplication process moves data to a chunk store and selectively compresses the data, replacing redundant copies of each chunk with a reference to a single copy. When this process is complete, the original files are replaced with reparse points that contain references to optimized data chunks.

Data deduplication is supported only on NTFS data volumes that are hosted on Windows Server 2012 R2 or Windows Server 2012 or Cluster Shared Volumes in Windows Server 2012 R2. Deduplication is not supported on boot volumes, FAT or ReFS volumes, remote mapped or remote mounted drives, CSV file system volumes in Windows Server 2012, live data (such as SQL Server databases), Exchange Server stores, or virtual machines that use local storage on a server running Hyper-V.

Files that are not supported include those with extended attributes, encrypted files, and files smaller than 32K. Files with reparse points will also not be processed. From a design perspective, deduplication is not supported for files that are open and constantly changing for extended periods or that have high I/O requirements, such as running virtual machines or live SQL Server databases. The exception is that in Windows Server 2012 R2, deduplication supports live VHDs for VDI for workloads.

7.5.2 Thin Provisioning and Trim

Like data deduplication, thin-provisioning technology improves the efficiency of how we use and provision storage. Instead of removing redundant data on the volume, thin provisioning gains efficiencies by making it possible to allocate just enough storage at the moment of storage allocation, and then increase capacity as your business needs grow over time.

Windows Server 2012 R2 provides full support for thinly provisioned storage arrays, which lets you get the most out of your storage infrastructure. These sophisticated storage solutions offer just-in-time allocations (known as thin provisioning), and the ability to reclaim storage that is no longer needed (known as trim).

7.5.3 Volume Cloning

Volume cloning is another common practice in virtualization environments. Volume cloning can be used for host and virtual machine volumes to dramatically improve provisioning times.

7.5.4 Volume Snapshot

SAN volume snapshots are a common method of providing a point-in-time, instantaneous backup of a SAN volume or LUN. These snapshots are typically block-level, and they only utilize storage capacity as blocks change on the originating volume. However, Windows Server does not control this behavior—it varies by storage array vendor.

7.5.5 Storage Tiering

Storage tiering physically partitions data into multiple distinct classes such as price or performance. Data can be dynamically moved among classes in a tiered storage implementation, based on access, activity, or other considerations.

Storage tiering normally combine varying types of disks that are used for different data types (for example, production, non-production, or backups).

7.6 Storage Management and Automation

Windows Server 2012 R2 provides a unified interface, which is a powerful and consistent mechanism for managing storage. The storage interface can reduce complexity and operational costs, and it provides capabilities for advanced management of storage, in addition to a core set of defined WMI and Windows PowerShell interfaces.

The interface uses WMI for comprehensive management of physical and virtual storage, including non-Microsoft intelligent storage subsystems, and it provides a rich experience for IT professionals and developers by using Windows PowerShell scripting to help make a diverse set of solutions available.

Management applications can use a single Windows API to manage storage types by using SMP or standards-based protocols such as Storage Management Initiative Specification (SMI-S). The WMI-based interface provides a single mechanism through which to manage all storage, including non-Microsoft intelligent storage subsystems and virtualized local storage (known as Storage Spaces).

Figure 13 shows the unified storage management architecture.

Figure 15 Unified storage management architecture

7.6.1 Windows Offloaded Data Transfers (ODX)

Whenever possible, the speed of your virtualization platform should rival that of physical hardware. Offloaded data transfers (ODX) is a feature in the storage stack in Windows Server 2012 R2. When used with offload-capable SAN storage hardware, ODX lets a storage device perform a file copy operation without the main processor of the server running Hyper-V reading the content from one storage place and writing it to another.

ODX enables rapid provisioning and migration of virtual machines, and it provides significantly faster transfers of large files, such as database or video files. By offloading the file transfer to the storage array, ODX minimizes latencies, promotes the use of array throughput, and reduces host resource usage such as central processing unit (CPU) and network consumption.

File transfers are automatically and transparently offloaded when you move or copy files, regardless of whether you perform drag-and-drop operations in Windows Explorer or use command-line file copy commands. No administrator setup or intervention is necessary.

8 Network Architecture

A variety of designs and new approaches to data center networks have emerged in recent years. The objective in most cases is to improve resiliency and performance while optimizing performance for highly virtualized environments.

8.1 Network Architecture Patterns

8.1.1 Hierarchical

Many network architectures include a hierarchical design with three or more tiers including the core tier, the aggregation (or distribution) tier, and the access tier. Designs are driven by the port bandwidth and quantity that are required at the edge, in addition to the ability of the core and aggregation tiers to provide higher speed uplinks to aggregate traffic. Additional considerations include Ethernet broadcast boundaries and limitations and loop-avoidance technologies.

Core tier

The core tier is the high-speed backbone for the network architecture. The core typically comprises two modular switch chassis to provide a variety of service and interface module options. The data entry of the core tier might interface with other network modules.

Aggregation tier

The aggregation (or distribution) tier consolidates connectivity from multiple access tier switch uplinks. This tier is commonly implemented in end-of-row switches, a centralized wiring closet, or main distribution frame room. The aggregation tier provides high-speed switching and more advanced features, like Layer 3 routing and other policy-based networking capabilities. The aggregation tier must have redundant, high-speed uplinks to the core tier for high availability.

Access tier

The access tier provides device connectivity to the data center network. This tier is commonly implemented by using Layer 2 Ethernet switches—typically through blade chassis switch modules or top-of-rack (ToR) switches. The access tier must provide redundant connectivity for devices, required port features, and adequate capacity for access (device) ports and uplink ports.

The access tier can also provide features that are related to NIC Teaming, such as Link Aggregation Control Protocol (LACP). Certain teaming solutions might require LACP switch features.

Figure 15 illustrates two three-tier network models—one provides 10 GbE to devices and the other provides 1 GbE to devices.

Figure 16 Comparative of gigabit Ethernet edge topology

8.1.2 Flat Network

A flat network topology is adequate for very small networks. In a flat network design, there is no hierarchy. Each networking device has essentially the same job, and the network is not divided into layers or modules. A flat network topology is easy to design and implement, and it is easy to maintain if the network stays small. When the network grows, however, a flat network is undesirable. The lack of hierarchy makes troubleshooting difficult because instead of being able to concentrate troubleshooting efforts in just one area of the network, you might have to inspect the entire network.

8.1.3 Network Virtualization (Software-Defined Networking)

Hyper-V Network Virtualization (HNV) supports the concept of a virtual network that is independent of the underlying physical network. With this concept of virtual networks (which are not the same as VPNs) which are composed of one or more virtual subnets, the exact physical location of an IP subnet is decoupled from the virtual network topology. As a result, customers can easily move their subnets to the cloud while preserving their existing IP addresses and topology in the cloud, so that existing services continue to work and they are unaware of the physical location of the subnets.

Hyper-V Network Virtualization in Windows Server 2012 R2 provides policy-based, software-controlled network virtualization that reduces the management overhead that enterprises face when they expand dedicated infrastructure-as-a-service (IaaS) clouds. In addition, it provides cloud hosting service providers with better flexibility and scalability for managing virtual machines so that they can achieve higher resource utilization.

8.2 Network Performance and Low Latency

8.2.1 Data Center Bridging

Separate isolated connections for network, live migration, and management traffic make managing network switches and other networking infrastructure a challenge. As data centers evolve, IT organizations look to some of the latest innovations in networking to help solve these issues. The introduction of 10 GbE networks, for example, helps support converged networks that can handle network, storage, live migration, and management traffic through a single connection, reducing the requirements for and costs of IT management.

Data-center bridging (DCB) refers to enhancements to Ethernet LANs that are used in data center environments. These enhancements consolidate the various forms of a network into a single technology, known as a converged network adapter or CNA. In the virtualized environment, Hyper-V in Windows Server 2012 R2 and Windows Server 2012 can utilize data-center bridging hardware to converge multiple types of network traffic on a single network adapter, with a maximum level of service to each.

Data-center bridging is a hardware mechanism that classifies and dispatches network traffic, supporting far fewer traffic flows. It converges various types of traffic, including network, storage, management, and live migration traffic. However, it also can classify network traffic that does not originate from the networking stack (for example, hardware that is accelerated through iSCSI when the system does not use the Microsoft iSCSI initiator).

8.2.2 Virtual Machine Queue (VMQ)

The virtual machine queue (VMQ) feature allows the network adapter of the host to pass DMA packets directly into individual virtual machine memory stacks. Each virtual machine device buffer is assigned a VMQ, which avoids needless packet copies and route lookups in the virtual switch. Essentially, VMQ allows the single network adapter of the host to appear as multiple network adapters to the virtual machines, to allow each virtual machine its own dedicated network adapter. The result is less data in the buffers of the host and an overall performance improvement in I/O operations.

The VMQ is a hardware virtualization technology that is used for the efficient transfer of network traffic to a virtualized host operating system. A VMQ-capable network adapter classifies incoming frames to be routed to a VMQ receive queue, based on filters that associate the queue with the network adapter of a virtual machine. These hardware queues can have affinities to specific CPUs to allow scaling on a per–virtual machine network adapter basis.

Windows Server 2012 R2 dynamically distributes the incoming network traffic to host processors, based on processor use and network load. In times of heavy network load, dynamic VMQ automatically uses more processors. In times of light network load, dynamic VMQ relinquishes those same processors.

Dynamic VMQ requires hardware network adapters and drivers that support Network Device Interface Specification (NDIS) 6.30 or higher.

8.2.3 IPsec Task Offload

IPsec protects network communication by authenticating and encrypting some or all of the content of network packets. IPsec Task Offload in Windows Server 2012 R2 utilizes the hardware capabilities of server network adapters to offload IPsec processing. This reduces the CPU overhead of IPsec encryption and decryption significantly.

In Windows Server 2012 R2, IPsec Task Offload is extended to virtual machines. Customers who use virtual machines and want to help protect their network traffic by using IPsec can utilize the IPsec hardware offload capability that is available in server network adapters. Doing so frees up CPU cycles to perform more application-level work and leaves the per-packet encryption and decryption to hardware.

8.2.4 Quality of Service (QoS)

Quality of Service (QoS) is a set of technologies that provide you with the ability to cost-effectively manage network traffic in network environments. There are three options for deploying QoS in Windows Server:

· Data center bridging. This is performed by hardware, and it is good for iSCSI environments. However, it requires hardware investments and it can be complex.

· Policy-based QoS. Historically present in Windows Server, this capability is managed with Group Policy. Challenges include that it does not provide the required capabilities within the Microsoft iSCSI initiator or Hyper-V environments.

· Hyper-V QoS. This capability works well for virtual machine workloads and virtual network adapters on servers running Hyper-V. However, it requires careful planning and an implementation strategy because it is not managed with Group Policy. (Networking is managed somewhat differently in Virtual Machine Manager.)

For most deployments, one or two 10 GbE network adapters should provide enough bandwidth for all the workloads on a server running Hyper-V. However, 10 GbE network adapters and switches are considerably more expensive than the 1 GbE counterparts. To optimize the 10 GbE hardware, a server running Hyper-V requires new functionality to manage bandwidth.

Windows Server 2012 R2 expands the power of QoS by providing the ability to assign a minimum bandwidth to a virtual machine or service. This feature is important for service providers and companies that honor SLA clauses that promise a minimum network bandwidth to customers. It is equally important to enterprises that require predictable network performance when they run virtualized server workloads on shared hardware.

In addition to the ability to enforce maximum bandwidth, QoS in Windows Server 2012 R2 provides a new bandwidth management feature: minimum bandwidth. Unlike maximum bandwidth, which is a bandwidth cap, minimum bandwidth is a bandwidth floor, and it assigns a certain amount of bandwidth to a specific type of traffic. It is possible to implement minimum and maximum bandwidth limits simultaneously.

8.2.5 Remote Direct Memory Access and SMB Direct

SMB Direct (SMB over RDMA) is a storage protocol in Windows Server 2012 R2. It enables direct memory-to-memory data transfers between server and storage, with minimal CPU usage, while using standard RDMA-capable network adapters. SMB Direct is supported on three types of RDMA technology: iWARP, InfiniBand, and RoCE. In Windows Server 2012 R2 there are more scenarios that can take advantage of RDMA connectivity including CSV redirected mode and live migration.

8.2.6 Receive Segment Coalescing

Receive segment coalescing improves the scalability of the servers by reducing the overhead for processing a large amount of network I/O traffic. It accomplishes this by coalescing multiple inbound packets into a large buffer.

8.2.7 Receive-Side Scaling

Receive-side scaling (RSS) spreads monitored interruptions over multiple processors, so a single processor is not required to handle all I/O interruptions, which was common in earlier versions of Windows Server. Active load balancing between the processors tracks the load on the CPUs and then transfers the interruptions as necessary.

You can select which processors will be used to handle RSS requests beyond 64 processors, which allows you to utilize high-end computers that have a large number of logical processors.

RSS works with NIC Teaming to remove a limitation in earlier versions of Windows Server, where you had to choose between using hardware drivers or RSS. RSS will also work for User Datagram Protocol (UDP) traffic, and it can manage and debug applications that use WMI and Windows PowerShell.

8.2.7.1 Virtual Receive-Side Scaling

Windows Server 2012 R2 includes support for virtual receive-side scaling, which much like standard RSS, allows virtual machines to distribute network processing loads across multiple virtual processors to increase network throughput within virtual machines.

Virtual receive-side scaling is only available on virtual machines running the Windows Server 2012 R2 and Windows 8.1 operating systems, and it requires VMQ support on the physical adapter. Virtual receive-side scaling is disabled by default if the VMQ-capable adapter is less than 10 Gbps.

In addition, SR-IOV cannot be enabled in a virtual machine network interface to take advantage of virtual receive-side scaling. Virtual receive-side scaling coexists with NIC Teaming, live migration, and Network Virtualization using Generic Routing Encapsulation (NVGRE).

8.2.8 SR-IOV

The SR-IOV standard was introduced by the Peripheral Component Special Interest Group (PCI-SIG). This group owns and manages the Peripheral Component Interconnect (PCI) specifications as open industry standards. SR-IOV works with system support for virtualization technologies that provides remapping for interrupts and DMA, and it lets SR-IOV–capable devices be assigned directly to a virtual machine.

Hyper-V in Windows Server 2012 R2 enables support for SR-IOV–capable network devices, and it allows the direct assignment of the SR-IOV virtual function of a physical network adapter to a virtual machine. This increases network throughput and reduces network latency, while reducing the host CPU overhead that is required for processing network traffic.

8.2.9 TCP Chimney Offload

The TCP chimney architecture offloads the data transfer portion of TCP protocol processing for one or more TCP connections to a network adapter. This architecture provides a direct connection, called a chimney, between applications and an offload-capable network adapter.

The chimney offload architecture reduces host network processing for network-intensive applications, so networked applications scale more efficiently and end-to-end latency is reduced. In addition, fewer servers are needed to host an application, and servers are able to use the full Ethernet bandwidth.

Note   Virtual machine chimney, also called TCP offload, was removed in Windows Server 2012 R2. The TCP chimney will not be available to guest operating systems because it applies only to host traffic.

8.3 Network High Availability and Resiliency

To increase reliability and performance in virtualized environments, Windows Server 2012 R2 and Windows Server 2012 include built-in support for network adapter hardware that is NIC Teaming–capable. NIC Teaming is also known as “load balancing and failover or LBFO”.

Not all traffic will benefit from NIC Teaming. The most noteworthy exception is storage traffic, where iSCSI should be handled by MPIO, and SMB should be backed by SMB Multichannel. However, when a single set of physical network adapters is used for storage and networking traffic, teaming for storage traffic is acceptable and encouraged.

8.3.1 NIC Teaming

NIC Teaming, allows multiple network adapters to be placed into a team for the purposes of bandwidth aggregation and traffic failover. This helps maintain connectivity in the event of a network component failure. NIC Teaming is included in Windows Server 2012 R2 and Windows Server 2012.

NIC Teaming is compatible with all networking capabilities in Windows Server 2012 R2 with five exceptions: SR-IOV, RDMA, Policy-Based QoS, TCP chimney, and 802.1X authentication. From a scalability perspective in Windows Server 2012 R2, a minimum of two and a maximum of 32 network adapters can be added to a single team, and an unlimited number of teams can be created on a single host.

8.3.1.1 NIC Teaming Types

Establishing NIC Teaming requires setting the teaming mode and distribution mode for the team. Two basic sets of algorithms are used for the teaming modes. These are exposed in the UI as three options—a switch-independent mode, and two switch-dependent modes: Static Teaming and Dynamic Teaming (LACP).

· Switch-independent mode: These algorithms make it possible for team members to connect to different switches because the switch does not know that the interface is part of a team on the host. These modes do not require the switch to participate in the teaming. This is generally recommended for Hyper-V deployments.

· Switch-dependent modes: These algorithms require the switch to participate in the teaming. All interfaces of the team are connected to the same switch.

There are two common choices for switch-dependent modes of NIC Teaming:

· Static teaming (based on IEEE 802.3ad): This mode requires configuration on the switch and on the host to identify which links form the team. Because this is a statically configured solution, there is no additional protocol to assist the switch and host to identify incorrectly plugged cables or other errors that could cause the team to fail. Typically, this mode is supported by server-class switches.

· Dynamic teaming (based on IEEE 802.1ax): This mode works by using the LACP to dynamically identify links that are connected between the host and a specific switch. Typical server-class switches support IEEE 802.1ax, but most require administration to enable LACP on the port. There are security challenges to allow an almost completely dynamic IEEE 802.1ax to operate on a switch. As a result, switches require that the switch administrator configure the switch ports that are allowed to be members of such a team.

Switch-dependent modes result in inbound and outbound traffic that approach the practical limits of the aggregated bandwidth.

8.3.1.2 Traffic-Distribution Algorithms

Aside from teaming modes, three algorithms are used for traffic distribution within NIC Teaming in Windows Server 2012 R2 and Windows Server 2012. These are exposed in the UI under the Load Balancing mode as Dynamic, Hyper-V Switch Port, and Address Hash.

· Dynamic: The Dynamic traffic distribution algorithm (sometimes referred to as adaptive load balancing) adjusts the distribution of load continuously in an attempt to more equitably carry the load across team members. This produces a higher probability that all the available bandwidth of the team can be used.
In this mode, NIC Teaming recognizes bursts of transmission for each flow, called a flowlet, and allows it to be redirected to a new network adapter for transmission. Outgoing traffic uses the address hash for distribution. On the receiving side, specific types of traffic (Address Resolution Protocol [ARP] and Internet Control Message Protocol [ICMP]) from a virtual machine is forced to encourage traffic to arrive on a particular network adapter. The remaining traffic is sent according to the address hash distribution.
This mechanism provides the benefits of multiple distribution schemes. This mode is particularly useful when virtual machine queues (VMQs) are used, and it is generally recommended for Hyper-V deployments where guest teaming is not enabled.

· Hyper-V Port: Used when virtual machines have independent MAC addresses that can be used as the basis for dividing traffic. There is an advantage in using this scheme in virtualization, because the adjacent switch always sees source MAC addresses on only one connected interface. This causes the switch to balance the egress load (the traffic from the switch to the host) on multiple links, based on the destination MAC address on the virtual machine.

Like Dynamic mode, this mode is particularly useful when virtual machine queues (VMQs) are used, because a queue can be placed on the specific network adapter where the traffic is expected to arrive. However, this mode might not be granular enough to get a well-balanced distribution, and it will always limit a single virtual machine to the bandwidth that is available on a single interface.

Windows Server uses the Hyper-V Switch Port as the identifier instead of the source MAC address, because a virtual machine in some instances might be using more than one MAC address.

· Address Hash: Creates a hash value that is based on components of the packet and then assigns packets that have that hash value to one of the available interfaces. This keeps all packets from the same TCP stream on the same interface. Components that can be used as inputs to the hashing function include:

· Source and destination MAC addresses

· Source and destination IP addresses, with or without considering the MAC addresses (2-tuple hash)

· Source and destination TCP ports, usually used with the IP addresses (4-tuple hash)

8.3.1.3 Guest Virtual Machine NIC Teaming

NIC Teaming in Windows Server 2012 R2 allows virtual machines to have virtual network adapters that are connected to more than one virtual switch and still have connectivity, even if the network adapter that is under that virtual switch is disconnected. This is particularly important when you are working with a feature such as SR-IOV traffic, which does not go through the Hyper-V virtual switch. Thus, it cannot be protected by a network adapter team that is under a virtual switch.

By using the virtual machine teaming option, you can set up two virtual switches, each of which is connected to its own SR-IOV–capable network adapter. NIC Teaming then works in one of the following ways:

· Each virtual machine can install a virtual function from one or both SR-IOV network adapters, and if a network adapter disconnection occurs, it will fail over from the primary virtual function to the backup virtual function.

· Each virtual machine can have a virtual function from one network adapter and a non-virtual function interface to the other switch. If the network adapter that is associated with the virtual function becomes disconnected, the traffic can fail over to the other switch without losing connectivity.

8.3.1.4 NIC Teaming Feature Compatibility

Information about the compatibility of the NIC Teaming feature in Windows Server 2012 R2 is provided in the following whitepaper on Microsoft TechNet: Windows Server 2012 R2 NIC Teaming (LBFO) Deployment and Management.

8.4 Network Isolation and Security

Windows Server 2012 R2 contains new security and isolation capabilities through the Hyper-V virtual switch. With Windows Server 2012 R2, you can configure servers running Hyper-V to enforce network isolation among any set of arbitrary isolation groups, which are typically defined for individual customers or sets of workloads.

Windows Server 2012 R2 provides isolation and security capabilities for multitenancy by offering the following new features:

· Multitenant virtual machine isolation through private virtual LANs (pVLANs)

· Protection from Address Resolution Protocol (ARP) and Neighbor Discovery protocol spoofing

· Protection against Dynamic Host Configuration Protocol (DHCP) spoofing with DHCP guard

· Isolation and metering by using virtual port access control lists (Port ACLs)

· The ability to use the Hyper-V virtual switch trunk mode to direct traffic from multiple VLANs to a single network adapter in a virtual machine

· Resource metering

· Windows PowerShell and Windows Management Instrumentation (WMI)

8.4.1 VLANs

Currently, VLANs are the mechanism that most organizations use to help support tenant isolation and reuse address space. A VLAN uses explicit tagging (VLAN ID) in the Ethernet frame headers, and it relies on Ethernet switches to enforce isolation and restrict traffic to network nodes that have the same VLAN ID.

8.4.2 Trunk Mode to Virtual Machines

With a VLAN, a set of host machines or virtual machines appear to be on the same LAN (i.e., the same physical network segment or collision domain), independent of their actual physical locations. By using the Hyper-V virtual switch trunk mode, traffic from multiple VLANs can be directed to a single network adapter in a virtual machine that could previously receive traffic from only one VLAN. As a result, traffic from different VLANs is consolidated, and a virtual machine can listen to multiple VLANs. This feature can help you shape network traffic and enforce multitenant security in your data center.

8.4.3 Private VLANs

VLAN technology is traditionally used to subdivide a network and provide isolation for individual groups that share a common physical infrastructure. Windows Server 2012 R2 introduces support for private VLANs, which is a technique that is used with VLANs that can be used to provide isolation between two virtual machines that are on the same VLAN. This could be useful in the following scenarios:

· Lack of free primary VLAN numbers in the data center or on physical switches. (There is a maximum of 4096, possibly less depending on the hardware that is used.)

· Isolating multiple tenants from each other in community VLANs while still providing centralized services (such as Internet routing) to all of them simultaneously (by using promiscuous mode).

When a virtual machine does not have to communicate with other virtual machines, you can use private VLANs to isolate it from other virtual machines that are in your data center. By assigning each virtual machine in a private VLAN only one primary VLAN ID and one or more secondary VLAN IDs, you can put the secondary private VLANs into one of three modes, as shown in the following table. These private VLAN modes determine which virtual machines on the private VLAN a virtual machine can talk to. To isolate a virtual machine, you should put it in isolated mode.

Private VLAN Mode	Description
Isolated	Isolated virtual machines cannot exchange packets with each other at Layer 2, and they cannot see each other. However, they can see virtual machines in promiscuous mode that are located in the same primary VLAN. There can be only one isolated secondary VLAN in a primary VLAN.
Promiscuous	Much like a traditional VLAN, virtual machines in promiscuous mode can exchange packets with any other virtual machine that is located on the same primary VLAN.
Community	Virtual machines in community mode that are on the same VLAN, can exchange packets with each other at Layer 2, and they can talk to other virtual machines in the same VLAN and to virtual machines that are in promiscuous mode. Virtual machines in community mode cannot talk to virtual machines that are in community mode or isolated mode.

Table 5 Private VLAN modes

8.4.4 ARP and Neighbor Discovery Spoofing Protection

The Hyper-V virtual switch helps provide protection against a malicious virtual machine stealing IP addresses from other virtual machines through ARP spoofing (also known as ARP poisoning in IPv4). This type of man-in-the-middle attack is known as Neighbor Discovery spoofing. A malicious virtual machine sends a fake ARP message, which associates its own MAC address to an IP address that it does not own.

Unsuspecting virtual machines send network traffic that is targeted to that IP address to the MAC address of the malicious virtual machine, instead of to the intended destination. For IPv6, Windows Server 2012 R2 helps provide equivalent protection for Neighbor Discovery spoofing. This is a mandatory scenario for hosting companies to consider when the virtual machine is not under control of the fabric or cloud administrators.

8.4.5 Router Guard

In Windows Server 2012 R2, the Hyper-V virtual switch helps protect against router advertisement and redirection messages that come from an unauthorized virtual machine pretending to be a router. In this situation, a malicious virtual machine attempts to be a router for other virtual machines. If a virtual machine accepts the network routing path, the malicious virtual machine can perform man-in-the-middle attacks, for example, steal passwords from SSL connections.

8.4.6 DHCP Guard

In a DHCP environment, a rogue DHCP server could intercept client DHCP requests and provide incorrect address information. The rogue DHCP server could cause traffic to be routed to a malicious intermediary that inspects all traffic before forwarding it to the legitimate destination.

To protect against this man-in-the-middle attack, the Hyper-V administrator can designate which Hyper-V virtual switches can have DHCP servers connected to them. DHCP server traffic from other Hyper-V virtual switches is automatically dropped. The Hyper-V virtual switch now helps protect against a rogue DHCP server that is attempting to provide IP addresses that would cause traffic to be rerouted.

8.4.7 Port ACLs

Port ACLs provide a mechanism for isolating networks and metering network traffic for a virtual port on the Hyper-V virtual switch. By using port ACLs, you can meter the IP addresses or MAC addresses that can (or cannot) communicate with a virtual machine. For example, you can use port ACLs to enforce isolation of a virtual machine by letting it talk only to the Internet, or communicate only with a predefined set of addresses.

By using the metering capability, you can measure network traffic that is going to or from a specific IP address or MAC address, which lets you report on traffic that is sent or received from the Internet or from network storage arrays.

You also can configure multiple port ACLs for a virtual port. Each port ACL consists of a source or destination network address, and permit, deny or meter action. The metering capability also supplies information about the number of instances that traffic was attempted to or from a virtual machine from a restricted (deny) address.

8.4.7.1 Virtual Switch Extended Port ACLs

In Windows Server 2012 R2, extended port ACLs can be configured on the Hyper-V virtual switch to allow and block network traffic to and from the virtual machines that are connected to a virtual switch through virtual network adapters. By default, Hyper-V allows virtual machines to communicate with each other when they are connected to the same virtual switch and the network traffic between virtual machines doesn’t leave the physical machine. In these cases, network traffic configurations on the physical network cannot manage traffic between virtual machines.

Service providers can greatly benefit from extended port ACLs because they can be used to enforce security policies between resources in the fabric infrastructure. The use of extended port ACLs is useful in multitenant environments, such as those provided by service providers. Tenants can also enforce security policies through extended port ACLs to isolate their resources.

In addition, virtualization has increased the number of security features. For example, port ACLs are required on physical top-of-rack switches due to the many-to-one nature of virtual machine connectivity. Extended port ACLs could potentially decrease the number of security policies that are required for the large number of servers in a service provider or large enterprise IaaS fabric infrastructure.

8.4.8 Network Virtualization

Isolating the virtual machines for different departments or customers can be a challenge on a shared network. When entire networks of virtual machines must be isolated, the challenge becomes even greater. Traditionally, VLANs have been used to isolate networks, but VLANs are very complex to manage on a large scale. The primary drawbacks of VLANs include:

· Cumbersome reconfiguration of production switches is required whenever virtual machines or isolation boundaries must be moved. Moreover, frequent reconfigurations of the physical network for the purposes of adding or modifying VLANs increases the risk of an outage.

· VLANs have limited scalability because typical switches support no more than 1,000 VLAN IDs (while the specification supports a maximum of 4,095).

· VLANs cannot span multiple Ethernet subnets, which limits the number of nodes in a single VLAN and restricts the placement of virtual machines based on physical location.

Windows Server 2012 R2 Hyper-V Network Virtualization (HNV) enables you to isolate network traffic from different business units or customers on a shared infrastructure, without having to use VLANs. HNV also lets you move virtual machines as needed within your infrastructure while preserving their virtual network assignments. You can even use network virtualization to transparently integrate these private networks into a preexisting infrastructure on another site.

HNV extends the concept of server virtualization to permit multiple virtual networks, potentially with overlapping IP addresses, to be deployed on the same physical network. By using Hyper-V Network Virtualization, you can set policies that isolate traffic in a dedicated virtual network, independently of the physical infrastructure.

To virtualize the network, HNV uses the following elements:

· Network Virtualization using Generic Routing Encapsulation (NVGRE)

· Policy management server (Virtual Machine Manager)

· Network Virtualization Gateway servers

The potential benefits of network virtualization include:

· Tenant network migration to the cloud with minimum reconfiguration or effect on isolation. Customers can keep their internal IP addresses while they move workloads onto shared clouds for IaaS, thus minimizing the configuration changes that are needed for IP addresses, DNS names, security policies, and virtual machine configurations.
In software-defined, policy-based data center networks, network traffic isolation does not depend on VLANs, but it is enforced within Hyper-V hosts, and based on multitenant isolation policies. Network administrators can use VLANs to manage traffic in the physical infrastructure if the topology is primarily static.

· Tenant virtual machine deployment anywhere in the data center. Services and workloads can be placed or migrated to any server in the data center while keeping their IP addresses, without being limited to a physical IP subnet hierarchy or VLAN configurations.

· Simplified network design and improved server and network resource use. The rigidity of VLANs and the dependency of virtual machine placement on a physical network infrastructure results in overprovisioning and underuse. By breaking this dependency, a virtual network increases the flexibility of virtual machine workload placement, thus simplifying network management and improving the use of servers and network resources.
Placement of server workloads is simplified because migration and placement of workloads are independent of the underlying physical network configurations. Server administrators can focus on managing services and servers, while network administrators can focus on overall network infrastructure and traffic management.

· Works with current hardware (servers, switches, appliances) to promote performance; however, network adapters that use NVGRE are recommended. Hyper-V Network Virtualization can be deployed in current data centers, and yet it is compatible with emerging data center “flat network” technologies such as Transparent Interconnection of Lots of Links (TRILL), which is an IETF-standard architecture that is intended to expand Ethernet topologies.

· Full management through Windows PowerShell and WMI. Although a policy management server such as System Center Virtual Machine Manager is highly recommended, it is possible to use Windows PowerShell to easily script and automate administrative tasks. Windows Server 2012 R2 includes Windows PowerShell cmdlets for network virtualization that let you build command-line tools or automated scripts to configure, monitor, and troubleshoot network isolation policies.

HNV in Windows Server 2012 R2 is implemented as part of the Hyper-V virtual switch (whereas, in Windows Server 2012 it was implemented as part of the NDIS filter driver). This allows forwarding extensions for Hyper-V virtual switch to co-exist with network virtualization configurations.

HNV also supports dynamic IP address learning, which allows network virtualization to learn about manually assigned or DHCP addresses to set on the virtual network. In environments that use System Center Virtual Machine Manager, when a host learns a new IP address, it will notify Virtual Machine Manager, which adds it to the centralized policy. This allows rapid dissemination and reduces the overhead that is associated with distributing the network virtualization routing policy. In Windows Server 2012 R2, HNV is also supported in configurations that also use NIC Teaming.

Windows Server 2012 R2 provides HNV Gateway server services to support site-to-site virtual private networks, Network Address Translation (NAT), and forwarding between physical locations to support multitenant hosting solutions that leverage network virtualization. This allows service providers and organizations that use HNV to support end-to-end communication from the corporate network or the Internet to the data center running HNV.

Without such gateway devices, virtual machines in a network are completely isolated from the outside, and they cannot communicate with non-network-virtualized systems such as other systems in the corporate network or on the Internet. In Windows Server 2012 R2 and Windows Server 2012, HNV Gateway can encapsulate and decapsulate NVGRE packets, based on the centralized network virtualization policy. It can perform the gateway-specific functionality on the resulting native CA packets, such as IP forwarding and routing, NAT, or site-to-site tunneling.

9 Compute Architecture

9.1 Server Architecture

The host server architecture is a critical component of the virtualized infrastructure, and a key variable in the consolidation ratio and cost analysis. The ability of the host server to handle the workload of a large number of consolidation candidates increases the consolidation ratio and helps provide the desired cost benefit.

The system architecture of the host server refers to the general category of the server hardware. Examples include rack mounted servers, blade servers, and large symmetric multiprocessor servers (SMP). The primary tenet to consider when selecting system architectures is that each server running Hyper-V will contain multiple guests with multiple workloads. Processor, RAM, storage, and network capacity are critical, as are high I/O capacity and low latency. The host server must be able to provide the required capacity in each of these categories.

Note   The Windows Server Catalog is useful for assisting customers in selecting appropriate hardware. It contains information about all servers, and storage and other hardware devices that are certified for Windows Server 2012 R2 and Hyper-V. The logo program and support policy for failover-cluster solutions changed in Windows Server 2012, and cluster solutions are not listed in the Windows Server Catalog. All individual components that make up a cluster configuration must earn the appropriate "Certified for" or "Supported on" Windows Server designations. These designations are listed in their device-specific category in the Windows Server Catalog.

9.1.1 Server and Blade Network Connectivity

Use multiple network adapters or multiport network adapters on each host server. For converged designs, network technologies that provide teaming or virtual network adapters can be utilized, provided that two or more physical adapters can be teamed for redundancy, and multiple virtual network adapters or VLANs can be presented to the hosts for traffic segmentation and bandwidth control.

9.1.2 Microsoft Multipath I/O

Multipath I/O (MPIO) architecture supports iSCSI, Fibre Channel, and SAS SAN connectivity by establishing multiple sessions or connections to the storage array.

Multipath solutions use redundant physical path components—adapters, cables, and switches—to create logical paths between the server and the storage device. If one or more of these components should fail (causing the path to fail), multipath logic uses an alternate path for I/O, so that applications can still access their data. Each network adapter (in the iSCSI case) or HBA should be connected by using redundant switch infrastructures, to provide continued access to storage in the event of a failure in a storage fabric component.

Failover times vary by storage vendor, and they can be configured by using timers in the Microsoft iSCSI Initiator driver or by modifying the parameter settings of the Fibre Channel host bus adapter driver.

In all cases, storage multipath solutions should be used. Generally, storage vendors will build a device-specific module on top of the MPIO software in Windows Server 2012 R2 or Windows Server 2012. Each device-specific module and HBA will have its own unique multipath options and recommended number of connections.

9.1.3 Consistent Device Naming (CDN)

Windows Server 2012 R2 supports consistent device naming (CDN), which provides the ability for hardware manufacturers to identify descriptive names of onboard network adapters within the BIOS. Windows Server 2012 R2 assigns these descriptive names to each interface, providing users with the ability to match chassis printed interface names with the network interfaces that are created within Windows. The specification for this change is outlined in the Slot Naming PCI-SIG Engineering Change Request.

9.2 Failover Clustering

9.2.1 Cluster-Aware Updating (CAU)

Cluster-Aware Updating (CAU) reduces server downtime and user disruption by allowing IT administrators to update clustered servers with little or no loss in availability when updates are performed on cluster nodes. CAU transparently takes one node of the cluster offline, installs the updates, performs a restart (if necessary), brings the node back online, and moves on to the next node. This feature is integrated into the existing Windows Update management infrastructure, and it can be further extended and automated with Windows PowerShell for integrating into larger IT automation initiatives.

CAU facilitates the cluster updating operation while running from a computer running Windows Server 2012 R2 or Windows 8.1. The computer running the CAU process is called an orchestrator (not to be confused with System Center Orchestrator). CAU supports two modes of operation: remote-updating mode or self-updating mode. In remote-updating mode, a computer that is remote from the cluster that is being updated acts as an orchestrator. In self-updating mode, one of the cluster nodes that is being updated acts as an orchestrator, and it is capable of self-updating the cluster on a user-defined schedule.

When using Cluster-Aware Updating, the end-to-end cluster update process by way of the CAU is cluster-aware, and it is completely automated. It integrates seamlessly with an existing Windows Update Agent (WUA) and Microsoft Windows Server Update Services (WSUS) infrastructure.

CAU also includes an extensible architecture that supports new plug-in development to orchestrate any node-updating tools, such as custom software installers, BIOS updating tools, and network adapter and HBA firmware updating tools. After they have been integrated with CAU, these tools can work across all cluster nodes in a cluster-aware manner.

9.2.2 Cluster Shared Volume (CSV)

The Cluster Shared Volumes (CSV) feature was introduced in Windows Server 2008 R2 as a more efficient way for administrators to deploy storage for cluster-enabled virtual machines on Hyper-V and other server roles, such as Scale-Out File Server and SQL Server.

Before CSV, administrators had to provision a LUN on shared storage for each virtual machine, so that each machine had exclusive access to its virtual hard disks, and mutual Write conditions could be avoided. By using CSV, all cluster hosts have simultaneous access to a single or multiple shared volumes where storage for multiple virtual machines can be hosted. Thus, there is no need to provision a new LUN whenever you created a new virtual machine.

Windows Server 2012 R2 provides the following CSV capabilities:

· Flexible application and file storage: Cluster Shared Volumes extends its potential benefits beyond Hyper-V to support other application workloads and flexible file storage solutions. CSV 2.0 provides capabilities to clusters through shared namespaces to share configurations across all cluster nodes, including the ability to build continuously available cluster-wide file systems. Application storage can be served from the same shared resource as data, eliminating the need to deploy two clusters (an application and a separate storage cluster) to support high availability application scenarios.

· Integration with other features in Windows Server 2012 R2: Allows for inexpensive scalability, reliability, and management simplicity through tight integration with Storage Spaces. You gain high performance and resiliency capabilities with SMB Direct and SMB Multichannel, and create more efficient storage with thin provisioning. In addition, Windows Server 2012 R2 supports ReFS, data deduplication, parity, tiered Storage Spaces, and Write-back cache in Storage Spaces.

· Single namespace: Provides a single consistent file namespace where files have the same name and path when viewed from any node in the cluster. CSV are exposed as directories and subdirectories under the ClusterStorage root directory.

· Improved backup and restore: Supports several backup and restore capabilities, including support for the full feature set of VSS and support for hardware and software backup of CSV. CSV also offers a distributed backup infrastructure for software snapshots. The Software Snapshot Provider coordinates to create a CSV 2.0 snapshot, point-in-time semantics at a cluster level, and the ability to perform remote snapshots.

· Optimized placement policies: CSV ownership is evenly distributed across the failover cluster nodes, based on the number of CSV that each node owns. Ownership is automatically rebalanced during conditions such as restart, failover, and the addition of cluster nodes.

· Increased resiliency: The SMB protocol is comprised of multiple instances per failover cluster node: a default instance that handles incoming traffic from SMB clients that access regular file shares, and a second CSV instance that handles only internode CSV metadata access and redirected I/O traffic between nodes. This improves the scalability of internode SMB traffic between CSV nodes.

· CSV Cache: Allows system memory (RAM) as Write-back cache. The CSV Cache provides caching of Read-only, unbuffered I/O, which can improve performance for applications that use unbuffered I/O when accessing files (for example, Hyper-V). CSV Cache delivers caching at the block level, which enables it to perform caching of pieces of data being accessed within the VHD file. CSV Cache reserves its cache from system memory and handles orchestration across the sets of nodes in the cluster. In Windows Server 2012 R2, CSV Cache is enabled by default and you can allocate up 80% of the total physical RAM for CSV Write-back cache.

There are several CSV deployment models, which are outlined in the following sections.

9.2.2.1 Single CSV per Cluster

In the “single CSV per cluster” design pattern, the SAN is configured to present a single large LUN to all the nodes in the host cluster. The LUN is configured as a CSV in failover clustering. All files that belong to the virtual machines that are hosted on the cluster are stored on the CSV. Optionally, data deduplication functionality that is provided by the SAN can be utilized (if it is supported by the SAN vendor).

Figure 17 Virtual machine on a single large CSV

9.2.2.2 Multiple CSV per Cluster

In the “multiple CSV per cluster” design pattern, the SAN is configured to present two or more large LUNs to all the nodes in the host cluster. The LUNs are configured as CSV in failover clustering. All virtual machine–related files are hosted on the cluster are stored on the CSV.

In addition, data deduplication functionality that the SAN provides can be utilized (if supported by the SAN vendor).

Figure 18 Virtual machines on multiple CSV with minimal segregation

For the single and multiple CSV patterns, each CSV has the same I/O characteristics, so that each individual virtual machine has all of its associated virtual hard disks (VHDs) stored on one CSV.

Figure 19 The virtual disks of each virtual machine reside on the same CSV

9.2.2.3 Multiple I/O Optimized CSV per Cluster

In the “multiple I/O optimized CSV per cluster” design pattern, the SAN is configured to present multiple LUNs to all the nodes in the host cluster; however, the LUNs are optimized for particular I/O patterns like fast sequential Read performance, or fast random Write performance. The LUNs are configured as CSV in failover clustering. All VHDs that belong to the virtual machines that are hosted on the cluster are stored on the CSV, but they are targeted to appropriate CSV for the given I/O needs.

Figure 20 Virtual machines with a high degree of virtual disk segregation

In the “multiple I/O optimized CSV per cluster” design pattern, each individual virtual machine has all of its associated VHDs stored on the appropriate CSV, per required I/O requirements.

Figure 21 Virtual machines with a high degree of virtual disk segregation

Note A single virtual machine can have multiple VHDs, and each VHD can be stored on different CSV (provided that all CSV are available to the host cluster on which the virtual machine is created).

9.2.2.4 BitLocker-Encrypted Cluster Volumes

Hyper-V, failover clustering, and BitLocker work together to create an ideal, highly secure platform for a private cloud infrastructure. Cluster disks that are encrypted with BitLocker Drive Encryption in Windows Server 2012 R2 enable better physical security for deployments outside secure data centers (if there is a critical safeguard for private cloud infrastructure) and help protect against data leaks.

9.3 Hyper-V Failover Clustering

A Hyper-V host failover cluster is a group of independent servers that work together to increase the availability of applications and services. The clustered servers are connected by physical cables and software. If one of the cluster nodes fails, another node begins to provide service—a process that is known as failover. In the case of a planned live migration, users will experience no perceptible service interruption.

The host servers are one critical component of a dynamic, virtual infrastructure. Consolidation of multiple workloads on the host servers requires that those servers have high availability. Windows Server 2012 R2 provides advances in failover clustering that enable high availability and live migration of virtual machines between physical nodes.

9.3.1 Host Failover-Cluster Topology

We recommend that the server topology consist of at least two Hyper-V host clusters. The first needs at least two nodes, and it is referred to as the fabric management cluster. The second, plus any additional clusters, is referred to as fabric host clusters.

In scenarios of smaller scale or specialized solutions, the management and fabric clusters can be consolidated in the fabric host cluster. Special care has to be taken to provide resource availability for the virtual machines that host the various parts of the management stack.

Each host cluster can contain up to 64 nodes. Host clusters require some form of shared storage such as a Storage Spaces, Scale-Out File Server cluster, Fibre Channel SAN, or iSCSI SAN.

9.3.2 Cluster Quorum and Witness Configurations

In quorum configurations, every cluster node has one vote, and a witness (disk or file share) also has one vote. A witness is recommended when the number of voting nodes is even, but it is not required when the number of voting nodes is odd. We always recommend that you keep an odd total number of votes in a cluster. Therefore, a cluster witness should be configured to support cluster configurations in Hyper-V when the number of failover cluster nodes is even.

Choices for a cluster witness include a shared disk witness and a file-share witness. There are distinct differences between these two models.

Disk witness   Consists of a dedicated LUN to serve as the quorum disk that is used as an arbitration point. A disk witness stores a copy of cluster database for all nodes to share. We recommend that this disk consist of a small partition that is at least 512 MB in size; however, it is commonly recommended to reserve a 1 GB disk for each cluster. This LUN can be NTFS- or ReFS-formatted, and it does not require the assignment of a drive letter.

File-share witness   Configurations use a simple, unique file share that is located on a file server to support one or more clusters. This file share must have Write permissions for the cluster name object (CNO), along with all of the nodes. We highly recommend that this file share exist outside any of the cluster nodes, and therefore, carry the requirement of additional physical or virtual servers outside the Hyper-V compute cluster within the fabric. Writing to this file share results in minimal network traffic, because all nodes contain separate copies of the cluster database, and only cluster membership changes are written to the file share.

The additional challenge that this creates is that file-share witness configurations are susceptible to “split” or “partition in time” scenarios, and it could create situations in which surviving nodes and starting nodes have different copies of the cluster database[2]. File-share witness disks should be used only in configurations in which no shared disk infrastructure exists.

Additional quorum and witness capabilities in Windows Server 2012 R2 include:

· Dynamic witness: By default Windows Server 2012 R2 clusters are configured to use dynamic quorum, which allows the witness vote to be dynamically adjusted and reduces the risk that the cluster will be impacted by witness failure. By using this configuration, the cluster decides whether to use the witness vote based on the number of voting nodes that are available in the cluster, which simplifies the witness configuration.
In addition, a Windows Server 2012 R2 cluster can dynamically adjust a running node’s vote to keep the total number of votes at an odd number, which allows the cluster to continue to run in the event of a 50% node split where neither side would normally have quorum.

· Force quorum resiliency: This change allows for the ability to force quorum in the event of a partitioned cluster. A partitioned cluster occurs when a cluster breaks into subnets that are not aware of each other and the service is restarted by forcing quorum.

9.3.3 Host Cluster Networks

A variety of host cluster networks are required for a Hyper-V failover cluster. The network requirements help enable high availability and high performance. The specific requirements and recommendations for network configuration are published in the TechNet Library in the Hyper-V: Live Migration Network Configuration Guide.

Note  the following list provides some examples, and it does not contain all network access types. For instance, some implementations would include a dedicated backup network.

Network Access Type	Purpose of the network-access type	Network-traffic requirements	Recommended network access
Storage	Access storage through SMB, iSCSI, or Fibre Channel. (Fibre Channel does not need a network adapter.)	High bandwidth and low latency.	Usually dedicated and private access. Refer to your storage vendor for guidelines.
Virtual machine access	Workloads that run on virtual machines usually require external network connectivity to service client requests.	Varies.	Public access, which could be teamed for link aggregation or to fail over the cluster.
Management	Managing the Hyper-V management operating system. This network is used by Hyper-V Manager or System Center Virtual Machine Manager (VMM).	Low bandwidth.	Public access, which could be teamed to fail over the cluster.
Cluster and Cluster Shared Volumes (CSV)	Preferred network that is used by the cluster for communications to maintain cluster health. Also used by CSV to send data between owner and non-owner nodes. If storage access is interrupted, this network is used to access the CSV or to maintain and back up the CSV. The cluster should have access to more than one network for communication to help make sure that it is a high availability cluster.	Usually low bandwidth and low latency. Occasionally high bandwidth.	Private access.
Live migration	Transfer virtual machine memory and state.	High bandwidth and low latency during migrations.	Private access.

Table 6 Network access types

9.3.3.1 Management Network

A dedicated management network is required so that hosts can be managed through a dedicated network to prevent competition with guest traffic needs. A dedicated network provides a degree of separation for the purposes of security and ease-of-management. A dedicated management network typically implies dedicating one network adapter for each host and port per network device to the management network.

Additionally, many server manufacturers provide a separate out-of-band (OOB) management capability that enables remote management of server hardware outside the host operating system.

9.3.3.2 iSCSI Network

If you are using iSCSI, a dedicated iSCSI network is required so that storage traffic is not in contention with any other traffic. This typically implies dedicating two network adapters for each host, and two ports per network device to the management network.

9.3.3.3 CSV/Cluster Communication Network

Usually, when the cluster node that owns a VHD file in a CSV performs disk I/O, the node communicates directly with the storage. However, storage connectivity failures sometimes prevent a given node from communicating directly with the storage. To maintain functionality until the failure is corrected, the node redirects the disk I/O through a cluster network (the preferred network for CSV) to the node where the disk is currently mounted. This is called CSV redirected I/O mode.

9.3.3.4 Live-Migration Network

During live migration, the contents of the memory of the virtual machine that is running on the source node must be transferred to the destination node over a LAN connection. To enable high-speed transfer, a dedicated live-migration network is required.

9.3.3.5 Virtual Machine Network(s)

The virtual machine networks are dedicated to virtual machine LAN traffic. A virtual machine network can be two or more 1 GbE networks, one or more networks that have been created through NIC Teaming, or virtual networks that have been created from shared 10 GbE network adapters.

9.3.4 Hyper-V Application Monitoring

With Windows Server 2012 R2, Hyper-V and failover clustering work together to bring higher availability to workloads that do not support clustering. They do so by providing a lightweight, simple solution to monitor applications that are running on virtual machines and by integrating with the host. By monitoring services and event logs inside the virtual machine, Hyper-V and failover clustering can detect if the key services that a virtual machine provides are healthy. If necessary, they provide automatic corrective action such as restarting the virtual machine or restarting a service within the virtual machine.

9.3.5 Virtual Machine Failover Prioritization

Virtual machine priorities can be configured to control the order in which specific virtual machines fail over or start. This helps make sure that high-priority virtual machines get the resources that they need and that lower-priority virtual machines are given resources as they become available.

9.3.6 Virtual Machine Anti-Affinity

Administrators can specify that two specific virtual machines cannot coexist on the same node in a failover scenario. By leveraging anti-affinity workloads, resiliency guidelines can be respected when they are hosted on a single failover cluster.

9.3.7 Virtual Machine Drain on Shutdown

Windows Server 2012 R2 supports shutting down a failover cluster node in Hyper-V without first putting the node into maintenance mode to drain any running clustered roles. The cluster automatically migrates all running virtual machines to another host before the computer shuts down.

9.3.8 Shared Virtual Hard Disk

Hyper-V in Windows Server 2012 R2 includes support for virtual machines to leverage shared VHDX files for shared storage scenarios such as guest clustering. Shared and non-shared virtual hard disk files that are attached as virtual SCSI disks appear as virtual SAS disks when you add a SCSI hard disk to a virtual machine.

10 Hyper-V Virtualization Architecture

10.1 Hyper-V Features

10.1.1 Hyper-V Host and Guest Scale-Up

Hyper-V in Windows Server 2012 R2 supports running on a host system that has up to 320 logical processors on hardware and 4 terabytes (TB) of physical memory. This helps encourage compatibility with the largest scale-up server systems.

Hyper-V in Windows Server 2012 R2 lets you configure a virtual machine with up to 64 virtual processors and up to 1 TB of memory, to support very large workload scenarios.

Hyper-V in Windows Server 2012 R2 also supports running up to 8,000 virtual machines on a 64-node failover cluster. This is a significant improvement over Hyper-V in Windows Server 2008 R2, which supported a maximum of 16 cluster nodes and 1,000 virtual machines per cluster.

10.1.2 Hyper-V over SMB 3.0

Prior to Windows Server 2012, remote storage options for Hyper-V were limited to expensive Fibre Channel SAN or iSCSI SAN solutions that were difficult to provision for Hyper-V guests, or more inexpensive options that did not offer many features. By enabling Hyper-V to use SMB file shares for VHDs, administrators have an option that is simple to provision with support for CSV,  inexpensive to deploy, and offers performance capabilities and features that rival those available with Fibre Channel SANs. As outlined earlier in the Storage Architecture section, SMB 3.0 can be leveraged for SQL Server and Hyper-V workloads. Hyper-V over SMB requires:

· One or more computers running Windows Server 2012 R2 with the Hyper-V and File and Storage Services roles installed.

· A common Active Directory infrastructure. (The servers that are running AD DS do not need to be running Windows Server 2012 R2.)

· Failover clustering on the Hyper-V side, on the File and Storage Services side, or both.

Hyper-V over SMB supports a variety of flexible configurations that offer several levels of capabilities and availability, which include single-node, dual-node, and multiple-node file server modes.

10.1.3 Virtual Machine Mobility

Hyper-V live migration makes it possible to move running virtual machines from one physical host to another with no effect on the availability of virtual machines to users. Hyper-V in Windows Server 2012 R2 introduces faster and simultaneous live migration inside or outside a clustered environment.

In addition to providing live migration in the most basic of deployments, this functionality facilitates more advanced scenarios, such as performing a live migration to a virtual machine between multiple, separate clusters to balance loads across an entire data center.

If you use live migration, you will see that live migrations can now use higher network bandwidths (up to 10 gigabits) to complete migrations faster. You can also perform multiple simultaneous live migrations to move many virtual machines quickly.

10.1.3.1 Hyper-V Live Migration

Hyper-V in Windows Server 2012 R2 lets you perform live migrations outside a failover cluster. The two scenarios for this are:

· Shared storage-based live migration. In this instance, the VHD of each virtual machine is stored on a local CSV or on a central SMB file share, and live migration occurs over TCP/IP or the SMB transport. You perform a live migration of the virtual machines from one server to another while their storage remains on the central local CSV or SMB file share.

· “Shared-nothing” live migration. In this case, the live migration of a virtual machine from one non-clustered Hyper-V host to another begins when the hard drive storage of the virtual machine is mirrored to the destination server over the network. Then you perform the live migration of the virtual machine to the destination server while it continues to run and provide network services.

In Windows Server 2012 R2, improvements have been made to live migration, including:

· Live migration compression. To reduce the total time of live migration on a system that is network constrained, Windows Server 2012 R2 provides performance improvements by compressing virtual machine memory data before it is sent across the network. This approach utilizes available CPU resources on the host to reduce the network load.

· SMB Multichannel. Systems that have multiple network connections between them can utilize multiple network adapters simultaneously to support live migration operations and achieve higher total throughput

· SMB Direct-based live migration. Live migrations that use SMB over RDMA-capable adapters use SMB Direct to transfer virtual machine data, which provides higher bandwidth, multichannel support, hardware encryption, and reduced CPU utilization during live migrations.

· Cross-version live migration. Live migration from servers running Hyper-V in Windows Server 2102 to Hyper-V in Windows Server 2012 R2 are supported to provide zero downtime. However, this only applies when you are migrating virtual machines from Hyper-V in Windows Server 2102 to Hyper-V in Windows Server 2012 R2, because down-level live migrations (moving from Hyper-V in Windows Server 2012 R2 to a previous version of Hyper-V) are not supported.

10.1.4 Storage Migration

Windows Server 2012 R2 supports live storage migration, which lets you move VHDs that are attached to a virtual machine that is running. When you have the flexibility to manage storage without affecting the availability of your virtual machine workloads, you can perform maintenance on storage subsystems, upgrade storage appliance firmware and software, and balance loads while the virtual machine is in use.

Windows Server 2012 R2 provides the flexibility to move VHDs on shared and non-shared storage subsystems if an SMB network shared folder on Windows Server 2012 R2 is visible to both Hyper-V hosts.

10.1.5 Hyper-V Virtual Switch

The Hyper-V virtual switch in Windows Server 2012 R2 is a Layer 2 virtual switch that provides programmatically managed and extensible capabilities to connect virtual machines to the physical network. The Hyper-V virtual switch is an open platform that lets multiple vendors provide extensions that are written to standard Windows API frameworks.

The reliability of extensions is strengthened through the Windows standard framework, and the required non-Microsoft code for functions is reduced. It is backed by the Windows Hardware Quality Labs certification program. You can manage the Hyper-V virtual switch and its extensions by using Windows PowerShell, or programmatically by using WMI or the Hyper-V Manager UI.

The Hyper-V virtual switch architecture in Windows Server 2012 R2 is an open framework that lets non-Microsoft vendors add new functionality such as monitoring, forwarding, and filtering into the virtual switch. Extensions are implemented by using Network Device Interface Specification (NDIS) filter drivers and Windows Filtering Platform (WFP) callout drivers. These public Windows platforms for extending Windows networking functionality are used as follows:

· NDIS filter drivers: Used to monitor or modify network packets in Windows. NDIS filters were introduced with the NDIS 6.0 specification.

· WFP callout drivers: Introduced in Windows Vista and Windows Server 2008, these drivers let independent software vendors (ISVs) create functionality to filter and modify TCP/IP packets, monitor or authorize connections, filter IPsec-protected traffic, and filter remote procedure calls (RPCs).
Filtering and modifying TCP/IP packets provides unprecedented access to the TCP/IP packet processing path. In this path, you can examine or modify outgoing and incoming packets before additional processing occurs. By accessing the TCP/IP processing path at different layers, you can more easily create firewalls, antivirus software, diagnostic software, and other types of applications and services.

The Hyper-V virtual switch is a module that runs in the root partition of Windows Server 2012 R2. The switch module can create multiple virtual switch extensions per host. All virtual switch policies—including QoS, VLAN, and ACLs—are configured per virtual network adapter.

Any policy that is configured on a virtual network adapter is preserved during a virtual machine state transition, such as a live migration. Each virtual switch port can connect to one virtual network adapter. In the case of an external virtual port, it can connect to a single physical network adapter on the Hyper-V server or to a team of physical network adapters.

The framework allows for non-Microsoft extensions to extend and affect the behavior of the Hyper-V virtual switch. The extensibility stack comprises an extension miniport driver and an extension protocol driver that are bound to the virtual switch. Switch extensions are lightweight filter drivers that bind between these drivers to form the extension stack.

There are three classes of extensions:

· Capture: Sit on top of the stack and monitor switch traffic.

· Filter: Sit in the middle of the stack and can monitor and modify switch traffic.

· Forwarding: Sit on the bottom of the stack and replace the virtual switch forwarding behavior.

Table 9 lists the various types of Hyper-V virtual switch extensions.

Feature	Purpose	Examples	Extensibility Component
Network Packet Inspection	Inspect network packets, but does not change them	sFlow and network monitoring	NDIS filter driver
Network Packet Filter	Inject, modify, and drop network packets	Security	NDIS filter driver
Network Forwarding	Non-Microsoft forwarding that bypasses default forwarding	Cisco Nexus 1000V, OpenFlow, Virtual Ethernet Port Aggregator (VEPA), and proprietary network fabrics	NDIS filter driver
Firewall/Intrusion Detection	Filter and modify TCP/IP packets, monitor or authoriz connections, filter IPsec-protected traffic, and filter RPCs	Virtual firewall and connection monitoring	WFP callout driver

Table 7 Windows Server 2012 R2 virtual switch extension types

Only one forwarding extension can be installed per virtual switch (overriding the default switching of the Hyper-V virtual switch), although multiple capture and filtering extensions can be installed. In addition, by monitoring extensions, you can gather statistical data by monitoring traffic at different layers of the switch. Multiple monitoring and filtering extensions can be supported at the ingress and egress portions of the Hyper-V virtual switch. Figure 21 shows the architecture of the Hyper-V virtual switch and the extensibility model.

Figure 22 Hyper-V extension layers

The Hyper-V virtual switch data path is bidirectional, which allows all extensions to see the traffic as it enters and exits the virtual switch. The NDIS send path is used as the ingress data path, while the NDIS receive path is used for egress traffic. Between ingress and egress, forwarding of traffic occurs by the Hyper-V virtual switch or by a forwarding extension.

Figure 22 outlines this interaction.

Figure 23 Hyper-V virtual switch bidirectional filter

Windows Server 2012 R2 provides Windows PowerShell cmdlets for the Hyper-V virtual switch that lets you build command-line tools or automated scripts for setup, configuration, monitoring, and troubleshooting. Windows PowerShell also helps non-Microsoft vendors build tools to manage their Hyper-V virtual switch.

10.1.5.1 Monitoring/Port Mirroring

Many physical switches can monitor the traffic that flows from specific ports through virtual machines. The Hyper-V virtual switch also provides port mirroring, which can help you designate which virtual ports should be monitored and to which virtual port the monitored traffic should be directed for further processing.

For example, a security-monitoring virtual machine can look for anomalous patterns in the traffic that flows through other virtual machines on the switch. In addition, you can diagnose network connectivity issues by monitoring traffic that is bound for a particular virtual switch port.

10.1.6 Virtual Fibre Channel

Windows Server 2012 R2 provides Fibre Channel ports within the Hyper-V guest operating system. This lets you connect to Fibre Channel directly from virtual machines when virtualized workloads have to connect to existing storage arrays. This protects your investments in Fibre Channel, lets you virtualize workloads that use direct access to Fibre Channel storage, lets you cluster guest operating systems over Fibre Channel, and offers an important new storage option for servers that are hosted in your virtualization infrastructure.

Fibre Channel in Hyper-V requires:

· One or more installations of Windows Server 2012 R2 with the Hyper-V role installed. Hyper-V requires a computer with processor support for hardware virtualization.

· A computer that has one or more Fibre Channel host bus adapters (HBAs), each of which has an updated HBA driver that supports virtual Fibre Channel. Updated HBA drivers are included with the HBA drivers for some models.

· Virtual machines that are configured to use a virtual Fibre Channel adapter, which must use Windows Server 2012 R2, Windows Server 2012, Windows Server 2008  R2, or Windows Server 2008 as the guest operating system.

· Connection only to data logical unit numbers (LUNs). Storage that is accessed through a virtual Fibre Channel that is connected to a LUN cannot be used as boot media.

Virtual Fibre Channel for Hyper-V provides the guest operating system with unmediated access to a SAN by using a standard World Wide Port Name (WWPN) that is associated with a virtual machine HBA. Hyper-V users can use Fibre Channel SANs to virtualize workloads that require direct access to SAN LUNs. Virtual Fibre Channel also allows you to operate in advanced scenarios, such as running the Windows failover clustering feature inside the guest operating system of a virtual machine that is connected to shared Fibre Channel storage.

Midrange and high-end storage arrays are capable of advanced storage functionality that helps offload certain management tasks from the hosts to the SANs. Virtual Fibre Channel presents an alternate hardware-based I/O path to the virtual hard disk stack in Windows software. This allows you to use the advanced functionality that is offered by your SANs directly from virtual machines running Hyper-V. For example, you can use Hyper-V to offload storage functionality (such as taking a snapshot of a LUN) on the SAN hardware by using a hardware Volume Shadow Copy Service (VSS) provider from within a virtual machine running Hyper-V.

Virtual Fibre Channel for Hyper-V guest operating systems uses the existing N_Port ID Virtualization (NPIV) T11 standard to map multiple virtual N_Port IDs to a single physical Fibre Channel N_Port. A new NPIV port is created on the host each time a virtual machine that is configured with a virtual HBA is started. When the virtual machine stops running on the host, the NPIV port is removed.

10.1.7 VHDX

Hyper-V in Windows Server 2012 R2 supports the updated VHD format (called VHDX) that has much larger capacity and built-in resiliency. The principal features of the VHDX format are:

· Support for virtual hard disk storage capacity of up to 64 TB

· Additional protection against data corruption during power failures by logging updates to the VHDX metadata structures

· Improved alignment of the virtual hard disk format to work well on large sector physical disks

The VHDX format also has the following features:

· Larger block sizes for dynamic and differential disks, which allows these disks to attune to the needs of the workload

· Four-kilobyte (4 KB) logical sector virtual hard disk that allows for increased performance when it is used by applications and workloads that are designed for 4 KB sectors

· Efficiency in representing data (called “trim”), which results in smaller files size and allows the underlying physical storage device to reclaim unused space. (Trim requires directly attached storage or SCSI disks and trim-compatible hardware.)

· The virtual hard disk size can be increased or decreased through the user interface while the virtual hard disk is in use if they are attached to a SCSI controller.

· Shared access by multiple virtual machines to support guest clustering scenarios.

10.1.8 Guest Non-Uniform Memory Access (NUMA)

Hyper-V in Windows Server 2012 R2 supports non-uniform memory access (NUMA) in a virtual machine. NUMA refers to a computer memory design that is used in multiprocessor systems in which the memory access time depends on the memory location relative to the processor.

By using NUMA, a processor can access local memory (memory that is attached directly to the processor) faster than it can access remote memory (memory that is local to another processor in the system). Modern operating systems and high-performance applications (such as SQL Server) have developed optimizations to recognize the NUMA topology of the system, and they consider NUMA when they schedule threads or allocate memory to increase performance.

Projecting a virtual NUMA topology in a virtual machine provides optimal performance and workload scalability in large virtual machine configurations. It does so by letting the guest operating system and applications (such as SQL Server) utilize their inherent NUMA performance optimizations. The default virtual NUMA topology that is projected into a Hyper-V virtual machine is optimized to match the NUMA topology of the host.

10.1.9 Dynamic Memory

Dynamic Memory, which was introduced in Windows Server 2008 R2 SP1, helps you use physical memory more efficiently. By using Dynamic Memory, Hyper-V treats memory as a shared resource that can be automatically reallocated among running virtual machines. Dynamic Memory adjusts the amount of memory that is available to a virtual machine, based on changes in memory demand and on the values that you specify.

In Windows Server 2012 R2, Dynamic Memory has a new configuration item called “minimum memory.” Minimum memory lets Hyper-V reclaim the unused memory from the virtual machines. This can result in increased virtual machine consolidation numbers, especially in VDI environments.

In Windows Server 2012 R2, Dynamic Memory for Hyper-V allows for the following:

· Configuration of a lower minimum memory for virtual machines to provide an effective restart experience.

· Increased maximum memory and decreased minimum memory on virtual machines that are running. You can also increase the memory buffer.

Windows Server 2012 introduced Hyper-V Smart Paging for robust restart of virtual machines. Although minimum memory increases virtual machine consolidation numbers, it also brings a challenge. If a virtual machine has a smaller amount of memory than its startup memory and it is restarted, Hyper-V needs additional memory to restart the virtual machine. Because of host memory pressure or the states of the virtual machines, Hyper-V might not always have additional memory available, which can cause sporadic virtual machine restart failures in Hyper-V environments. Hyper-V Smart Paging bridges the memory gap between minimum memory and startup memory to let virtual machines restart more reliably.

10.1.10 Hyper-V Replica

You can use failover clustering to create high availability virtual machines, but this does not protect your company from an entire data center outage if you do not use hardware-based SAN replication across your data centers. Hyper-V Replica fills an important need by providing an affordable failure recovery solution for an entire site, down to a single virtual machine.

Hyper-V Replica is closely integrated with Windows failover clustering, and it provides asynchronous, unlimited replication of your virtual machines over a network link from one Hyper-V host at a primary site to another Hyper-V host at a replica site, without relying on storage arrays or other software replication technologies.

Hyper-V Replica in Windows Server 2012 R2 supports replication between a source and target server running Hyper-V, and it can support extending replication from the target server to a third server (known as extended replication). The servers can be physically co-located or geographically separated.

Hyper-V Replica tracks the Write operations on the primary virtual machine and replicates these changes to the replica server efficiently over a WAN. The network connection between servers uses the HTTP or HTTPS protocol.

Hyper-V Replica supports integrated and certificate-based authentication. Connections that are configured to use integrated authentication are not encrypted. For an encrypted connection, use certificate-based authentication. Hyper-V Replica provides support for replication frequencies of 15 minutes, 5 minutes, or 30 seconds.

In Windows Server 2012 R2, you can access recovery points up to 24-hours old. (Windows Server 2012 allowed access to recovery points up to 15-hours old.)

10.1.11 Resource Metering

Hyper-V in Windows Server 2012 R2 supports resource metering, which is a technology that helps you track historical data on the use of virtual machines and gain insight into the resource use of specific servers. In Windows Server 2012 R2, resource metering provides accountability for CPU and network usage, memory, and storage IOPS.

You can use this data to perform capacity planning, monitor consumption by business units or customers, or capture data that is necessary to help redistribute the costs of running a workload. You could also use the information that this feature provides to help build a billing solution, so that you can charge customers of your hosting services appropriately for their usage of resources.

10.1.12 Enhanced Session Mode

Enhanced session mode allows redirection of devices, clipboard, and printers from clients that are using the Virtual Machine Connection tool. The enhanced session mode connection uses a Remote Desktop Connection session through the virtual machine bus (VMBus), which does not require an active network connection to the virtual machine as would be required for a traditional Remote Desktop Protocol connection.

Enhanced session mode connections provide additional capabilities beyond simple mouse, keyboard, and monitor redirection. It supports capabilities that are typically associated with Remote Desktop Protocol sessions, such as display configuration, audio devices, printers, clipboards, smart card, USB devices, drives, and supported Plug and Play devices.

Enhanced session mode requires a supported guest operating system such as Windows Server 2012 R2 or Windows 8.1, and it may require additional configuration inside the virtual machine. By default, enhanced session mode is disabled in Windows Server 2012 R2, but it can be enabled through the Hyper-V server settings.

10.2 Hyper-V Guest Virtual Machine Design

Standardization is a key tenet of private cloud architectures and virtual machines. A standardized collection of virtual machine templates can drive predictable performance and greatly improve capacity planning capabilities. As an example, the following table illustrates the composition of a basic virtual machine template library. Note that this should not replace proper template planning by using System Center Virtual Machine Manager.

Template	Specifications	Network	Operating system	Unit cost
Template 1—Small	2 vCPU, 4 GB memory, 50 GB disk	VLAN 20	Windows Server 2008 R2	1
Template 2—Medium	8 vCPU, 16 GB memory, 100 GB disk	VLAN 20	Windows Server 2012 R2	2
Template 3— X-Large	24 vCPU, 64 GB memory, 200 GB disk	VLAN 20	Windows Server 2012 R2	4

Table 8 Example template specification

10.2.1 Virtual Machine Storage

10.2.1.1 Dynamically Expanding Virtual Hard Disks

Dynamically expanding VHDs provide storage capacity as needed to store data. The size of the VHD file is small when the disk is created, and it grows as data is added to the disk. The size of the VHD file does not shrink automatically when data is deleted from the virtual hard disk; however, you can use the Edit Virtual Hard Disk Wizard to make the disk more compact and decrease the file size after data is deleted.

10.2.1.2 Fixed-Size Disks

Fixed-size VHDs provide storage capacity by using a VHDX file that is in the size that is specified for the virtual hard disk when the disk is created. The size of the VHDX file remains fixed, regardless of the amount of data that is stored. However, you can use the Edit Virtual Hard Disk Wizard to increase the size of the VHDX, which in turn increases the size of the VHDX file. By allocating the full capacity at the time of creation, fragmentation at the host level is not an issue. (Fragmentation inside the VHDX itself must be managed within the guest.)

10.2.1.3 Differencing Virtual Hard Disks

Differencing VHDs provide storage to help you make changes to a parent VHD without changing the disk. The size of the VHD file for a differencing disk grows as changes are stored to the disk.

10.2.1.4 Pass-Through Disks

Pass-through disks allow Hyper-V virtual machine guests to directly access local disks or SAN LUNs that are attached to the physical server, without requiring the volume to be presented to the host server.

The virtual machine guest accesses the disk directly (by using the GUID of the disk) without having to utilize the file system of the host. Given that the performance difference between fixed disk and pass-through disks is now negligible, the decision is based on manageability. For instance, a disk that is formatted with VHDX is hardly portable if the data on the volume will be very large (hundreds of gigabytes), given the extreme amounts of time it takes to copy. For a backup scheme with pass-through disks, the data can be backed up only from within the guest.

When you are utilizing pass-through disks, no VHDX file is created because the LUN is used directly by the guest. Because there is no VHDX file, and there is no dynamic sizing or snapshot capability.

Pass-through disks are not subject to migration with the virtual machine, and they are only applicable to standalone Hyper-V configurations. Therefore, they are not recommended for common, large scale Hyper-V configurations. In many cases, virtual Fibre Channel capabilities supersede the need to leverage pass-through disks when you are working with Fibre Channel SANs.

10.2.1.5 In-guest iSCSI Initiator

Hyper-V can also utilize iSCSI storage by directly connecting to iSCSI LUNs that are utilizing the virtual network adapters of the guest. This is mainly used for access to large volumes on SANs to which the Hyper-V host is not connected, or for guest clustering. Guests cannot boot from iSCSI LUNs that are accessed through the virtual network adapters without utilizing an iSCSI initiator.

10.2.1.6 Storage Quality of Service

The storage Quality of Service (QoS) in Windows Server 2012 R2 provides the ability to specify a maximum I/O operations per second (IOPS) value for a virtual hard disk in Hyper-V. This protects tenants in a multitenant Hyper-V infrastructure from consuming excessive storage resources, which can impact other tenants (often referred to as a “noisy neighbor”).

Maximum and minimum values are specified in terms of normalized IOPS where every 8 K of data is counted as an I/O. Administrators can configure storage QoS for a virtual hard disk that is attached to a virtual machine. Configurations include setting maximum and minimum values for the IOPS for the virtual hard disk. Maximum values are enforced and minimum values generate administrative notifications.

10.2.2 Virtual Machine Networking

Hyper-V guests support two types of virtual switch adapters: synthetic and emulated.

Synthetic adapters  Make use of the Hyper-V VMBus architecture. They are high-performance, native devices. Synthetic devices require that the Hyper-V integration services are installed in the guest operating system.

Emulated adapters  Available to all guests, even if integration services are not available. They perform much more slowly, and they should be used only if synthetic devices are unavailable.

You can create many virtual networks on the server running Hyper-V to provide a variety of communications channels. For example, you can create virtual switches to provide the following types of communication:

· Private network. Provides communication between virtual machines only.

· Internal network. Provides communication between the host server and virtual machines.

· External network. Provides communication between a virtual machine and a physical network by creating an association to a physical network adapter on the host server.

10.2.3 Virtual Machine Compute

10.2.3.1 Virtual Machine Generation

In Windows Server 2012 R2, Hyper-V virtual machine generation is a new feature that determines the virtual hardware and functionality that is presented to the virtual machine. Hyper-V in Windows Server 2012 R2 supports the following two generation types:

Generation 1 virtual machines   These represent the same virtual hardware and functionality that was available in previous versions of Hyper-V, which provides BIOS-based firmware and a series of emulated devices.

Generation 2 virtual machines   These provide the platform capability to support advanced virtual features. Generation 2 virtual machines deprecate legacy BIOS and emulators from virtual machines, which can potentially increase security and performance. Generation 2 virtual machines provide a simplified virtual hardware model including Unified Extensible Firmware Interface (UEFI) firmware.

Several legacy devices are removed from generation 2 virtual machines. Generation 2 virtual machines have a simplified virtual hardware model provided through the VMBus as synthetic devices (such as networking, storage, video, and graphics). This model supports Secure Boot, boot from a SCSI virtual hard disk or virtual DVD, or PXE boot by using a standard network adapter. In addition, generation 2 virtual machines support Unified Extensible Firmware Interface (UEFI) firmware.

Support for several legacy devices is also removed from generation 2 virtual machines, including Integrated Drive Electronics (IDE) and legacy network adapters. Generation 2 virtual machines are only supported in Windows Server 2012 R2, Windows Server 2012, Windows 8.1, and Windows 8 (64-bit versions).

10.2.3.2 Virtual Processors and Fabric Density

Providing sufficient CPU density is a matter of identifying the required number of physical cores to serve as virtual processors for the fabric. When performing capacity planning, it is proper to plan against cores and not hyperthreads or symmetric multithreads (SMT). Although SMT can boost performance (approximately 10 to 20 percent), SMT is not equivalent to cores.

A minimum of two CPU sockets is required for product line architecture (PLA) pattern configurations. Combined with a minimum 6-core CPU model, 12 physical cores per scale unit host are available to the virtualization layer of the fabric. Most modern server class processors support a minimum of six cores, with some supporting up to 10 cores. Given the average virtual machine requirement of two virtual CPUs, a two-socket server that has the midrange of six-core CPUs provides 12 logical CPUs. This provides a potential density of between 96 and 192 virtual CPUs on a single host.

As an example, Table 13 outlines the estimated virtual machine density based on a two-socket, six-core processor that uses a predetermined processor ratio. Note that CPU ration assumptions are highly dependent on workload analysis and planning and should be factored into any calculations. For this example, the processor ratio would have been defined through workload testing, and an estimation of potential density and required reserve capacity could then be calculated.

Nodes	Sockets	Cores	Total Cores	Logical CPU	pCPU/vCPU Ratio	Available Virtual CPU	Average Virtual CPU Workload	Estimated Raw Virtual Machine Density	Virtual Machine Density Less Reserve Capacity
1	2	6	12	12	8	96	2	48	N/A
4	2	6	12	48	8	384	2	192	144
8	2	6	12	96	8	768	2	384	336
12	2	6	12	144	8	1152	2	576	528
16	2	6	12	192	8	1536	2	768	720
32	2	6	12	384	8	3072	2	1536	1344
64	2	6	12	768	8	6144	2	3072	2688
1	2	8	16	16	8	128	2	64	N/A
4	2	8	16	64	8	512	2	256	192
8	2	8	16	128	8	1024	2	512	448
12	2	8	16	192	8	1536	2	768	704
16	2	8	16	256	8	2048	2	1024	960
32	2	8	16	512	8	4096	2	2048	1792
64	2	8	16	1024	8	8192	2	4096	3584
1	2	10	20	20	8	160	2	80	N/A
4	2	10	20	80	8	640	2	320	240
8	2	10	20	160	8	1280	2	640	560
12	2	10	20	240	8	1920	2	960	880
16	2	10	20	320	8	2560	2	1280	1200
32	2	10	20	640	8	5120	2	2560	2240
64	2	10	20	1280	8	10240	2	5120	4480

Table 9 Example virtual machine density chart

There are also supported numbers of virtual processors in a specific Hyper-V guest operating system. For more information, please see Hyper-V Overview on Microsoft TechNet.

10.2.4 Linux-Based Virtual Machines

Linux virtual machine support in Windows Server 2012 R2 and System Center 2012 R2 is focused on providing more parity with Windows-based virtual machines. The Linux Integration Services are now part of the Linux kernel version 3.x.9 and higher. This means that it is no longer a requirement to install Linux Integration Services as part of the image build process. For more information, please see The Linux Kernel Archives.

Linux virtual machines can natively take advantage of the following:

· Dynamic Memory. Linux virtual machines can take advantage of the increased density and resource usage efficiency of Dynamic Memory. Memory ballooning and hot add are supported.

· Support for online snapshots. Linux virtual machines can be backed up live, and they have consistent memory and file systems. Note that this is not app-aware consistency similar to what would be achieved with Volume Shadow Copy Service in Windows.

· Online resizing of VHDs. Linux virtual machines can expand VHDs in cases where the underlying LUNs are resized.

· Synthetic 2D frame buffer driver. Improves display performance within graphical apps.

This can be performed on Windows Server 2012 or Windows Server 2012 R2 Hyper-V.

10.2.5 Automatic Virtual Machine Activation

Automatic Virtual Machine Activation is a new feature in Windows Server 2012 R2 that makes it easier for service providers and large IT organizations to use the licensing advantages that are offered through the Windows Server 2012 R2 Datacenter edition—specifically, unlimited virtual machine instances for a licensed system.

Automatic Virtual Machine Activation allows customers to simply license the server, which provides as many resources as desired and as many virtual machine instances of Windows Server as the system can deliver. When guests are running Windows Server 2012 R2 and hosts are running Windows Server 2012 R2 Datacenter, guests automatically activate. This makes it easier and faster to deploy cloud solutions based on Windows Server 2012 R2 Datacenter.

Automatic Virtual Machine Activation (AVMA) requires that the host operating system (running on the bare metal) is running Hyper-V in Windows Server 2012 R2 Datacenter. The supported guest operating systems (running in the virtual machine) are:

· Windows Server 2012 R2 Standard

· Windows Server 2012 R2 Datacenter

· Windows Server 2012 R2 Essentials

11 Microsoft Azure and IaaS Architecture

Microsoft Azure is the Microsoft platform for the public cloud. You can use this platform in many ways, for instance:

· Build a web application that runs and stores its data in Microsoft data centers.

· Store data and the applications that use this data on-premises (that is, outside the public cloud).

· Create virtual machines for development, and then test or run production deployments of SharePoint and other applications.

· Build massive, scalable applications that have thousands or millions of users.

For a detailed description of the Microsoft Azure services, see Microsoft Azure Solutions.

Microsoft Azure provides platform as a service (PaaS), infrastructure as a service (IaaS), and Microsoft Azure virtual machines. With the IaaS capability, Microsoft Azure becomes a core part of the Microsoft Cloud OS vision. To create hybrid cloud architectures, it is critical that you have a deep understanding of the services provided in and the architecture of Microsoft Azure (shown in the following image).

Figure 24 Microsoft Azure poster

Note   To download a full-size version of this poster, see Microsoft Azure Poster in the Microsoft Download Center.

11.1 Microsoft Azure Services

11.1.1 Microsoft Azure Compute Services

Virtual Machines

Microsoft Azure Virtual Machines enable you to deploy a Windows Server or Linux image in the cloud. You can select images from a gallery or provide your customized images.

Cloud Services

Microsoft Azure Cloud Services remove the need to manage server infrastructure. With Web and Worker roles, Cloud Services enable you to quickly build, deploy and manage modern applications.

Web Sites

Microsoft Azure Web Sites enables you to deploy web applications on a scalable and reliable cloud infrastructure. You can quickly scale up or scale automatically to meet your application needs.

Mobile Services

Microsoft Azure Mobile Services provides a scalable cloud backend for building applications for Windows Store, Windows Phone, Apple iOS, Android, and HTML or JavaScript. You can store data in the cloud, authenticate users, and send push notifications to your application within minutes.

11.1.2 Microsoft Azure Data Services

Storage

Microsoft Azure Storage offers non-relational data storage including Blob, Table, Queue and Drive storage.

SQL Database

Microsoft Azure SQL Database is a relational database service that enables you to rapidly create, extend, and scale relational applications into the cloud.

SQL Reporting

Microsoft Azure SQL Reporting allows you to build easily accessible reporting capabilities into your Microsoft Azure application. You can get up and running in hours versus days at a lower upfront cost without the hassle of maintaining your own reporting infrastructure.

Backup

Microsoft Azure Backup manages cloud backups through familiar tools in Windows Server 2012 R2, Windows Server 2012 R2 Essentials, or Data Protection Manager in System Center 2012 R2 or System Center 2012.

Cache

Microsoft Azure Cache is a distributed, in-memory, scalable solution that enables you to build highly scalable and responsive applications by providing fast access to data.

HDInsight

Microsoft Azure HDInsight is a service that is based on Apache Hadoop. It allows you to manage, analyze, and report on structured or unstructured data.

Hyper-V Recovery Manager

Microsoft Azure Hyper-V Recovery Manager is a service that uses the Microsoft Azure public cloud to orchestrate and manage replication of your primary data center to a secondary site for the purposes of data management and continuity, and disaster recovery. The service makes it possible for you to use off-premises automation to control on-premises private clouds that are defined in Microsoft System Center 2012 R2 or System Center 2012 Service Pack 1 (SP1) Virtual Machine Manager.

11.1.3 Microsoft Azure Network Services

Virtual Network

Microsoft Azure Virtual Network enables you to create virtual private networks (VPNs) within Microsoft Azure and securely link them to other virtual networks or to an on-premises networking infrastructure.

Traffic Manager

Microsoft Azure Traffic Manager allows you to load balance incoming traffic across multiple hosted Microsoft Azure services, whether they’re running in the same data center or across data centers around the world.

11.1.4 Microsoft Azure Application Services

Active Directory

Microsoft Azure Active Directory provides identity management and access control capabilities for your cloud applications. You can synchronize your on-premises identities and enable single sign-on to simplify user access to cloud applications.

Multi-Factor Authentication

Microsoft Azure Multi-Factor Authentication helps prevent unauthorized access to on-premises and cloud applications by providing an additional layer of authentication. You can follow organizational security and compliance standards while also addressing user demand for convenient access.

Service Bus

Microsoft Azure Service Bus is a messaging infrastructure that sits between applications and allows them to exchange messages for improved scale and resiliency.

Notification Hubs

Microsoft Azure Notification Hubs provide a highly scalable, cross-platform push notification infrastructure that enables you to broadcast push notifications to millions of users at once or tailor notifications to individual users.

BizTalk Services

Microsoft Azure BizTalk Services is a powerful and extensible cloud-based integration service that provides business-to-business and enterprise application integration capabilities for delivering cloud and hybrid integration solutions within a secure, dedicated, per-tenant environment.

Media Services

Microsoft Azure Media Services allows you to create workflows for the creation, management, and distribution of digital media. It offers cloud-based media solutions from existing technologies including encoding, format conversion, content protection, and on-demand and live streaming.

The Microsoft Azure Training Kit provides a robust set of training and documentation. For more information, see:

· Microsoft Azure Training Kit website

· Microsoft Azure Training Kit presentations

· Microsoft Azure Training Kit demos

11.2 Microsoft Azure Accounts and Subscriptions

A Microsoft Azure subscription grants you access to Microsoft Azure services and to the Microsoft Azure Management Portal. The terms of the Microsoft Azure account determine the scope of activities that you can perform in the Management Portal and describe the limits on available storage, network, and compute resources. For more information, please see:

· Manage Accounts, Subscriptions, and Administrative Roles

· Microsoft Azure Account Portal

In the Management Portal, you see only the services that are created by using a subscription for which you are an administrator. The billing account sets the number of compute units (virtual machines), hosted services, and storage that can be used. You can view usage information for a service by clicking the name of the service in the Management Portal.

A Microsoft Azure subscription has two aspects:

· The Microsoft Azure account, through which resource usage is reported and services are billed. Each account is identified by a Microsoft Account or a corporate email account, and it is associated with at least one subscription. The account owner monitors usage and manages billings through the Microsoft Azure Account Center.

· The subscription, which governs access to and use of the Microsoft Azure subscribed service. The subscription holder uses the Management Portal to manage services.

The account and the subscription can be managed by the same individual or by different individuals or groups. In a corporate enrollment, an account owner might create multiple subscriptions to give members of the technical staff access to services.

Because resource usage within an account billing is reported for each subscription, an organization can use subscriptions to track expenses for projects, departments, regional offices, and so on. The account owners use the Microsoft account that is associated with the account to sign in to the Microsoft Azure Account Center. Individuals will not have access to the Management Portal unless they have created a subscription.

Subscriptions that are created through a corporate enrollment are based on credentials that the organization provides. The subscription holder (who uses the services, but is not responsible for billings) has access to the Management Portal but not to the Microsoft Azure Account Center. By contrast, the personal account holder (who performs both duties) can sign in to either portal by using the Microsoft account that is associated with the account.

By default, Microsoft Azure subscriptions have the following boundaries:

· 20 storage accounts (default)

· 200 terabytes (TB) per storage account

· 50 virtual machines in service

· 25 PaaS roles in service (default)

· 20 cloud services per subscription (default)

· 250 endpoints per cloud service

· 1,024 virtual machines in a virtual network

11.2.1 Sharing Service Management by Adding Co-Administrators

When a Microsoft Azure subscription is created, a service administrator is assigned. The default service administrator is the contact person for the subscription. For an individual subscription, this is the person who holds the Microsoft account that identifies the subscription. The Microsoft Azure account owner can assign a different service administrator by editing the subscription in the Microsoft Azure Account Center.

The service administrator for a subscription has full administrator rights to all Microsoft Azure services that are subscribed to and all hosted services that are deployed under the subscription. The service administrator also can perform administrative tasks for the subscription itself in the Management Portal. For example, the service administrator can manage storage accounts, affinity groups, and management certificates for the subscription.

To share management of hosted services, the service administrator can add co-administrators to the subscription. To be added as a co-administrator, a person needs only a Windows Live ID.

Subscription co-administrators share the same administrator rights as the service administrator, with one exception: a co-administrator cannot remove the service administrator from a subscription. Only the Microsoft Azure account owner can change the service administrator for a subscription by editing the subscription in the Microsoft Azure Account Center.

Important: Because service administrators and co-administrators in Microsoft Azure have broad Administrator rights for Microsoft Azure services, you should assign strong passwords for the Microsoft accounts that identify the subscribers and ensure that the credentials are not shared with unauthorized users.

In the Management Portal, the enterprise account owner only has the rights that are granted to any subscription holder. To sign in to the Management Portal, the account owner must be an administrator for a subscription. When account owners sign in to the Management Portal, they can see and manage only those hosted services that have been created under subscriptions for which they are an administrator. Enterprise account owners cannot see hosted services for subscriptions that they create for other people. To gain visibility into service management under subscriptions that they create, enterprise account owners can ask the subscription holders to add them as co-administrators.

11.2.2 Manage Storage Accounts for Your Subscription

You can add storage accounts to a Microsoft Azure subscription to provide access to data management services in Microsoft Azure. The storage account represents the highest level of the namespace for accessing each of the storage service components, including Blob services, Queue services, and Table services. Each storage account provides access to storage in a specific geographic region or affinity group.

11.2.3 Create Affinity Groups to Use with Storage Accounts and Hosted Services

By using affinity groups, you can co-locate storage and hosted services within the same data center. To use an affinity group with a hosted service, assign an affinity group instead of a geographic region when you create the service. The same option is available when you create a storage account. You cannot change the affinity group for an existing hosted service or storage account.

11.2.4 Add Management Certificates to a Microsoft Azure Subscription

Management certificates enable client access to Microsoft Azure resources for using the Microsoft Azure SDK tools, the Microsoft Azure Tools for Microsoft Visual Studio, or the Microsoft Azure Service Management REST APIs. For example, a management certificate is used to authenticate a user who is using Visual Studio tools to create and manage hosted services or using Windows PowerShell or command-line tools to deploy virtual machine role images.

Management certificates are not required when you work in the Management Portal. In the Management Portal, authentication is performed by using the credentials of the administrator who is performing the operation.

11.2.5 Creating and Managing Microsoft Azure Environments

Getting started with Microsoft Azure is relatively straightforward on an individual or small-business basis. However, for enterprise scenarios, proper management and utilization of the preceding constructs is critical from security, administration, and billing standpoints. As an example, the following diagram illustrates a simple scenario in which the Finance department utilizes a combination of billing accounts, Microsoft Azure subscriptions, Microsoft Azure Service Administrators, and Microsoft Azure Co-Administrators to model development, test, and production environments in Microsoft Azure.

Figure 25 Microsoft Azure poster

During creation of Microsoft Azure environments for medium- and large-size organizations, careful planning of the billing and administration scope is required. The preceding diagram outlines an example of how to model an organization that wants centralized billing, but with different organizational units, to manage and track its Microsoft Azure usage.

It is important to realize that in many areas, a Microsoft Azure subscription and the resources (networks, virtual machines, and so on) within that subscription have access boundaries. Communication between resources in two different subscriptions is not possible except by configuring publicly accessible endpoints or utilizing Microsoft Azure virtual private network (VPN) functionality to connect the on-premises data center to each subscription and routing traffic between the two. (This will be explained in more detail throughout this document.)

Related to billing accounts and subscriptions, administrator and co-administrator management is also a key consideration in designing Microsoft Azure environments. Although the use of a Microsoft account ID represents the default scenario, there are also options for federating an on-premises instance of Active Directory with Microsoft Azure Active Directory for a variety of scenarios, including managing administrative access to Microsoft Azure subscriptions.

If the customer uses an on-premises directory service, you can integrate it with their Microsoft Azure Active Directory to automate cloud-based administrative tasks and provide users with a more streamlined sign-in experience. Microsoft Azure Active Directory supports the following directory integration capabilities:

· Directory synchronization: Used to synchronize on-premises directory objects (such as users, groups, and contacts) to the cloud to help reduce administrative overhead.
After directory synchronization has been set up, administrators can provision directory objects from the on-premises instance of Active Directory to their tenant.

· Single sign-on (SSO): Used to provide users with a more seamless authentication experience as they access Microsoft cloud services while they are signed in to the corporate network. To set up SSO, organizations must deploy a security-token service on premises. After SSO is set up, users can use their Active Directory corporate credentials (user name and password) to access the services in the cloud and in their existing on-premises resources.

11.3 Microsoft Azure Service-Level Agreements

For the most up-to-date information about Microsoft Azure service-level agreements (SLAs), see Service Level Agreements on the Microsoft Azure Support site.

11.3.1 Caching

Microsoft defines SLAs to customers for connectivity between the caching endpoints and our Internet gateway. SLA calculations are based on an average, over a monthly billing cycle, by using five-minute time intervals. For more information, see
Caching SLAs on the Microsoft Download Center.

11.3.2 Content Delivery Network (CDN)

Microsoft guarantees that at least 99.9% of the time Microsoft Azure Content Delivery Network (CDN) will respond to client requests and deliver the requested content without error. We will review and accept data from any commercially reasonable independent measurement system that you choose to monitor your content. You must select a set of agents from the measurement system’s list of standard agents that are generally available and represent at least five geographically diverse locations in major worldwide metropolitan areas (excluding People’s Republic of China). For more information, see Microsoft Azure Content Distribution Network Service Level Agreement on the Microsoft Download Center.

11.3.3 Cloud Services, Virtual Machines, and Virtual Network

For Cloud Services, when you deploy two or more role instances in different fault and upgrade domains, Internet facing roles will have external connectivity as defined by the Azure SLA.

For all Internet facing virtual machines that have two or more instances deployed in the same availability set, Microsoft guarantees you will have external connectivity at least 99.95% of the time.

For Virtual Network, Microsoft provides Virtual Network Gateway server availability as defined by the Azure SLA. For more information, see Microsoft Azure Cloud Services, Virtual Machines, and Virtual Network SLA on the Microsoft Download Center.

11.3.4 Media Services

Availability of REST API transactions for media services encoding is defined by the Azure SLA. On-demand streaming will successfully make service requests with a 99.9% availability guarantee for existing media content when at least one On-Demand Streaming Reserved Unit is purchased. Availability is calculated over a monthly billing cycle. For more information, see Microsoft Azure Media Services Service Level Agreement on the Microsoft Download Center.

11.3.5 Mobile Services

Availability of REST API calls to all provisioned Microsoft Azure Mobile Services running in Standard and Premium tiers in a customer subscription is defined by the Azure SLA. No SLA is provided for the free tier of Mobile Services. Availability is calculated over a monthly billing cycle. For more information, see Microsoft Azure Mobile Services SLA on the Microsoft Download Center.

11.3.6 Multi-Factor Authentication

Availability of Microsoft Azure Multi-Factor Authentication is defined by the Azure SLA. The service is considered unavailable when it is unable to receive or process authentication requests for the multi-factor authentication provider that is deployed in a customer’s subscription. Availability is calculated over a monthly billing cycle. For more information, see Multi-Factor Authentication SLA on the Microsoft Download Center.

11.3.7 Service Bus

Microsoft Azure Service Bus Relays are defined by the Azure SLA and properly configured applications will be able to establish a connection to a deployed Relay.

For Service Bus Queues and Service Bus Topics, availability of properly configured applications will be able to send or receive messages or perform other operations on a deployed Queue or Service Bus Topic as defined by the Azure SLA.

Availability of properly configured applications will be able to send notifications or perform registration management operations with respect to a Service Bus Notification Hub deployed within a Basic or Standard Notification Hub As defined by the Azure SLA. For more information, see Service Bus SLA on the Microsoft Download Center.

11.3.8 SQL Server Database

SQL Server database customers will maintain connectivity between the database and the Azure Internet gateway at a “monthly availability” of 99.9% during a billing month. The monthly availability percentage for a specific customer database is the ratio of the time the database was available to customer to the total time in the billing month. Time is measured in five-minute intervals in a 30-day monthly cycle. Availability is always calculated for a full billing month. An interval is marked as unavailable if the customer’s attempts to connect to a database are rejected by the SQL Server database gateway. For more information, see SQL Database Service Level Agreement on the Microsoft Download Center.

11.3.9 SQL Server Reporting Services

SQL Server Reporting Services will maintain a “monthly availability” of 99.9% during a billing month. The monthly availability percentage is the ratio of the total time the customer’s SQL Server Reporting Services instances were available to the total time the instances were deployed in the billing month. Time is measured in five-minute intervals. Availability is always calculated for a full billing month. An interval is marked as unavailable if the customer’s initiated attempts to upload, execute, or delete reports fail to ever complete due to circumstances within Microsoft’s control. For more information, see SQL Reporting Service Level Agreement on the Microsoft Download Center.

11.3.10 Storage

Microsoft will successfully process correctly formatted requests that we receive to add, update, read, and delete data as defined by the Azure SLA. We also guarantee that your storage accounts will have connectivity to our Internet gateway. For more information, see Microsoft Azure Storage Service Level Agreement on the Microsoft Download Center.

11.3.11 Web Sites

Microsoft Azure Web Sites running in the Standard tier will respond to client requests 99.9% of the time for a given billing month. Monthly availability is calculated as the ratio of the total time the customer’s Standard websites were available to the total time the websites were deployed in the billing month. The website is deemed unavailable if the website fails to respond due to circumstances within Microsoft’s control. For more information, see Microsoft Azure Web Sites SLA on the Microsoft Download Center.

11.4 Microsoft Azure Pricing

Microsoft Azure pricing is based on usage, and it has different metrics and pricing depending on the Microsoft Azure services or resources (storage, virtual machines, and so on) that are used. There are different payment models, from pay-as-you-go to prepaid plans. For details about pricing and pricing calculators, see Microsoft Azure pricing at-a-glance

11.5 Extending the Data Center Fabric to Microsoft Azure

As customers move to a hybrid cloud architecture, a primary scenario is extending their data center fabric (compute, storage, or network) to the cloud. There are technical (burst capacity, backup, disaster recovery, and so on) and financial (usage-based costing) reasons for doing so.

Extending the fabric to the cloud can be performed in a number of ways, such as utilizing Microsoft Azure, a Microsoft hosting partner, or a non-Microsoft cloud services (such as a service from Amazon or Google).

The two approaches presented in this document are applicable to Microsoft Azure or Microsoft hosting partners.

11.5.1 Data Management Services in Microsoft Azure

This section provides information about using data management services in Microsoft Azure to store and access data. Data management services in Microsoft Azure consist of:

· Blob service: Stores large amounts of data that can be accessed from anywhere via HTTP or HTTPS

· Queue service: Stores large numbers of messages that can be accessed via HTTP or HTTPS

· Table service: Stores large amounts of structured, non-relational data

In addition, Microsoft Azure drives are a capability of the Blob service in that acts as an NTFS volume that is mounted on a role instance’s file system and is accessible to code running in a cloud service deployment.

StorSimple provides a hybrid cloud storage solution that integrates on-premises storage with data management services in Microsoft Azure. Additionally, multiple Microsoft partners and independent software vendors (ISVs) such as CommVault and STEALTH deliver solutions that integrate with Microsoft Azure.

11.5.1.1 Data Management Services

Data management services in Microsoft Azure underlie all of the PaaS and IaaS storage needs in Microsoft Azure. Although the data management services in Microsoft Azure are typically used in PaaS scenarios by application developers, they are relevant to IaaS, because all Microsoft Azure disks and images (including virtual machine VHD files) utilize the underlying data management services in Microsoft Azure, such as the Blob service and storage accounts.

The Microsoft Azure Blob service stores large amounts of unstructured data that can be accessed from anywhere in the world via HTTP or HTTPS. A single blob can be hundreds of gigabytes in size, and a single storage account can contain up to 100 TB of blobs. Common uses for the Blob service include:

· Serving images or documents directly to a browser.

· Storing files for distributed access.

· Streaming video and audio.

· Performing secure backup and disaster recovery.

· Storing data for analysis by an on-premises service or a service hosted by Microsoft Azure.

You can use the Blob service to expose data publicly to the world or privately for internal application storage. The Blob service contains the following components:

· Storage account: All access to Microsoft Azure Storage is done through a storage account. This is the highest level of the namespace for accessing blobs. An account can contain an unlimited number of containers, but their total size must be under 100 TB.

· Container: A container provides a grouping for a set of blobs. All blobs must be in a container. An account can contain an unlimited number of containers. A container can store an unlimited number of blobs within the 100 TB storage-account limit.

· Blob: A file of any type and size within the overall size limits that are outlined in this section. You can store two types of blobs in Microsoft Azure Storage: block blobs and page blobs. Most files are block blobs. A single block blob can be up to 200 GB in size. Page blobs can be up to 1 TB in size, and they are more efficient when ranges of bytes in a file are modified frequently. For more information about blobs, see Migrating Data to Microsoft Azure Blob Storage.

· URL format: Blobs are addressable by using the following URL format:

https://<storage account>.blob.core.windows.net/<container>/<blob>

For a detailed view of the Microsoft Azure Storage service architecture, the following resources are highly recommended:

Microsoft Azure Storage: A Highly Available Cloud Storage Service with Strong Consistency

· Paper

· Slides

· Recording

Data management services in Microsoft Azure are critical in a hybrid cloud scenario because all data that is stored in Microsoft Azure, including IaaS virtual machine VHD files, utilizes the underlying data management services in Microsoft Azure, which distribute data across multiple disks and data centers that are transparent to the running virtual machines.

Individual storage accounts have the following scalability targets:

· Capacity: Up to 200 TB

· Transactions: Up to 20,000 entities, messages, and blobs per second

· Bandwidth for a geo-replication storage account

· Ingress: Up to 5 Gbps

· Egress: Up to 10 Gbps

· Bandwidth for a local storage resource

· Ingress: Up to 10 Gbps

· Egress: Up to 15 Gbps

Note  The actual transaction and bandwidth targets that are achieved by your storage account will very much depend on the size of objects, access patterns, and the type of workload that your application exhibits. To go above these targets, a service should be built to use multiple storage accounts and partition the blob containers, tables, queues, and objects across those storage accounts. By default, a single Microsoft Azure subscription comes with 20 storage accounts. However, you can contact customer support to add more storage accounts if you have to store more data—for example, if you need petabytes of data.

Planning the usage of storage accounts for deployed virtual machines and services is a key design consideration.

Virtual Hard Disks

In Microsoft Azure Storage, drives, disks, and images are all virtual hard disks (VHDs) that are stored as page blobs within your storage account. There are several slightly different VHD formats: fixed, dynamic, and differencing. Currently, Microsoft Azure supports only the fixed format.

This format lays out the logical disk linearly within the file format, so that disk offset X is stored at blob offset X. At the end of the blob, a small footer describes the properties of the VHD. All of this, which is stored in the page blob, adheres to the standard VHD format, so that you can take this VHD and mount it on your server on-premises if you choose to.

Often, the fixed format wastes space, because most disks have large unused ranges in them. However, fixed VHDs are stored as a page blob, which is a sparse format, which provides the benefits of fixed and expandable disks at the same time.

Figure 26 Azure storage

Storage Replication

Data management services in Microsoft Azure provide several options for data redundancy and replication. Some are enabled by default and included in the base pricing. For more information, see Storage Pricing Details.

Locally Redundant Storage

Locally redundant storage provides highly durable and available storage within a single location. Microsoft Azure maintains an equivalent of three copies (replicas) of your data within the primary location. For more information, see Symposium on Operating Systems Principles.

This ensures that Microsoft Azure can recover from common failures (disk, node, rack) without affecting your storage account’s availability and durability. All storage writes are performed synchronously across three replicas in three separate fault domains before success is returned to the client. If there was a major data center disaster, in which part of a data center was lost, Microsoft would contact customers about potential data loss for locally redundant storage by using the customer’s subscription contact information.

Geo Redundant Storage

Geo redundant storage provides Microsoft Azure’s highest level of durability by additionally storing your data in a second location within the same region hundreds of miles away from the primary location. All Microsoft Azure Blob and Table data is geo-replicated; however, Queue data is not geo-replicated at this time.

With geo redundant storage, Microsoft Azure maintains three copies (replicas) of the data in the primary location and the secondary location. This ensures that each data center can recover from common failures on its own and provides a geo-replicated copy of the data in the event of a major disaster.

As in locally redundant storage, data updates are committed to the primary location before success is returned to the client. After this is complete, these updates are geo-replicated asynchronously to the secondary location.

Primary Region	Secondary Region
North Central US	South Central US
South Central US	North Central US
East US	West US
West US	East US
North Europe	West Europe
West Europe	North Europe
South East Asia	East Asia
East Asia	South East Asia

Geo redundant storage is enabled by default for all storage accounts that are in production. You can choose to disable this default state by turning off geo-replication in the Microsoft Azure portal for your accounts. You can also configure your redundant storage option when you create a new account through the Microsoft Azure Portal. For further details, see Introducing Geo-replication for Microsoft Azure Storage.

Important  Geo-redundant storage is not compatible or supported when operating system disk striping is utilized. For example, if you are using sixteen 1 TB disks in a virtual machine and using operating system striping to create a single 16 TB volume in the virtual machine, the storage must be locally redundant only. For more information, see Getting Started with SQL Server in Microsoft Azure Virtual Machines.

11.5.1.2 StorSimple

StorSimple, a Microsoft company, is the leading vendor of cloud-integrated storage for Microsoft Azure. StorSimple solutions combine the data-management functions of primary storage, backup, archive, and disaster recovery with Microsoft Azure integration, enabling customers to optimize storage costs, data protection, and service agility. With its unique cloud-snapshot capability, StorSimple automatically protects and rapidly restores production data by using Microsoft Azure Storage.

The StorSimple solution combines a number of storage technologies, including Internet SCSI (iSCSI) storage area network (SAN), snapshot, backup, deduplication, and compression with storage services that are offered by cloud service providers. StorSimple solutions seamlessly integrate advanced SAN technologies (such as SSDs, SAS, automated storage tiering, deduplication, compression, and encryption) with cloud storage to reduce the storage footprint significantly and lower capital expenditures (CapEx) and operating expenditures (OpEx).

StorSimple provides a strong solution for storage tiers that is integrated tightly with Microsoft Azure Storage. StorSimple optimizes usage of different cost or performance storage options that are based automatically on data usage. This opens a large number of architecture scenarios.

Note StorSimple is currently a hardware storage appliance that is available from Microsoft. The appliance acts as an iSCSI Target Server for on-premises servers and provides several tiers of on-premises storage (by using SSD and SAS) while natively integrating with Microsoft Azure Storage.

StorSimple consolidates primary storage, archive, backup, and disaster recovery through seamless integration with the cloud. By combining StorSimple software and a custom-designed enterprise-grade hardware platform, StorSimple solutions provide high performance for primary storage and enable revolutionary speed, simplicity, and reliability for backup and recovery. Note that in the figure below that CiS refers to cloud integrated storage.

Figure 27 Hybrid storage

· Infrastructure consolidation. StorSimple solutions consolidate primary storage, archives, backup, and disaster recovery through seamless integration with the cloud.

· Simpler, faster backup and recovery. StorSimple cloud-based snapshots enable revolutionary speed, simplicity, and reliability for backup and recovery. Users can achieve up to 100 times faster data recovery versus traditional backup methods used in the cloud.

· Secure data storage. StorSimple applies AES-256 encryption for all data that is transferred and stored in the cloud by using a private key that is known only to the organization.

· Lower overall storage costs. By integrating the cloud with local enterprise storage, StorSimple can reduce total storage costs through the use of Microsoft Azure storage.

StorSimple solutions use cloud storage as an automated storage tier, offloading capacity-management burdens and ongoing capital costs. Using local and cloud snapshots, application-consistent backups complete in a fraction of the time that traditional backup systems require, while reducing the amount of data that is transferred and stored in the cloud.

Cloud-based and location-independent disaster recovery allows customers to recover their data from virtually any location that has an Internet connection and test their disaster recovery plans without affecting production systems and applications. Thin provisioning from data in the cloud enables users to resume operations after a disaster much faster than possible with physical or cloud-based tape.

Appliance Model*	5020	7020	5520	7520
Capacity
Usable local hard-drive capacity	2 TB	4 TB	10 TB	20 TB
SSD (Enterprise MLC [eMLC]) physical capacity	400 GB	600 GB	1.2 TB	2 TB
Effective local capacity**	4–10 TB	8–20 TB	20–50 TB	40–100 TB
Maximum capacity	100 TB	200 TB	300 TB	500 TB
High Availability Features
Dual, redundant, hot-swappable power-cooling modules (PCMs)	2 × 764 W PCMs, 100–240 VAC	2 × 764 W and 2 × 580 W PCMs, 100–240 VAC
Network interfaces	4 × 1 gigabit per second (Gbps) copper
Controllers	Dual, redundant, hot-swappable, active, or hot-standby controllers with automatic failover
RAID protection	Yes, including SAS hot-spare
Storage Features
iSCSI with multipath I/O support	Yes
Primary data reduction	Yes
Acceleration	Nonvolatile random access memory, SSD, cloud storage acceleration
Microsoft certification	Windows Hardware Quality Labs
VMware certification	Yes, VMware vSphere versions 4.1 and 5.1
Support for VMware vStorage APIs for array integration	Yes. (Pending future certification)
Automatic storage tiers	SSD, SAS, and cloud storage
Adaptive I/O processing	Yes. Optimizes I/O performance of mixed-pattern workloads
Data portability	Yes, access data sets across StorSimple appliances
Data Protection Features
Local backups	Yes, by using snapshots
Offsite backups or tape elimination	Yes, by using cloud snapshots and cloud clones
Microsoft VSS application consistent backups	Yes, by using Data Protection Console and hardware VSS provider
Windows Cluster Shared Volumes (CSV) and dynamic disk support	Yes. Backup CSV, mirrored dynamic disks, multi-partition disks
Protected storage migration	Yes, by using Windows host-side mirroring. Allows online backups and nondisruptive cutover
Security Features
Virtual private storage	Yes
Data-in-motion encryption	HTTPS/Secure Socket Layer (SSL)
Data-at-rest encryption	AES-256-CBC
Volume access control	IQN, CHAP
Additional security features	Multiple user accounts (local and Active Directory), role-based access, secure web proxy support
Manageability and Serviceability
Nondisruptive software upgrade	Yes, updates and new releases
Hot-swappable components	Controllers, power and cooling modules, nonvolatile random access memory batteries, SSD and SAS drives
Management and monitoring	Integrated web GUI, email alerts with call home capability, Simple Network Management Protocol (SNMP) v1/v2c
Hardware Footprint
Form factor	2U rack mountable appliance	4U rack mountable appliance
Dimensions (L × W × H [in inches])	24.8” × 19” × 3.46”	24.8” × 19” × 6.96”

The StorSimple solution uses the industry-standard iSCSI SAN protocol to connect to servers. iSCSI is easily configured for use with Microsoft and VMware servers, and it is widely understood by storage administrators.

StorSimple is intended to be used as the primary storage solution for enterprise tier 2 applications, including email, file shares, Microsoft SharePoint, content management systems, virtual machines, and large unstructured data repositories. It is not built for latency sensitive applications such as online transaction processing.

The StorSimple solution uses three different types of storage: performance-oriented flash SSDs, capacity-oriented SAS disk drives, and cloud storage. Data is moved from one type of storage to another according to its relative activity level and customer-chosen policies. Data that becomes more active is moved to a faster type of storage and data that becomes less active is moved to a higher capacity type of storage.

There are four logical tiers in the system—two at the SSD level and one each in the SAS and cloud storage levels.

Tier Name	Storage Type	Data Activity	Reduction Applied
Native	SSD	New, most active	None
Hot	SSD	Existing, most active	Deduplication
Warm	SAS	Between hot and cool	Full
Cool	Cloud	Least active	Full

The fourth column in the preceding table indicates the type of data-reduction technology that is used in the various tiers. The native tier has none, the hot tier uses data deduplication, and the warm and cool tiers use full reduction, which means that data is both compressed and deduplicated. Notice that the progression from native tier to warm tier implies that data is deduplicated before it is compressed.

Data deduplication reduces the amount of data that is stored in the system by identifying data duplicates and removing excess copies. Data deduplication is particularly effective in virtual server environments. Compression reduces the amount of data that is stored in the system by identifying strings of repeated data values and replacing them with encoded values.

Another capacity conserving technology in the StorSimple solution is thin provisioning, which allocates storage capacity as it is needed, as opposed to reserving capacity in advance. All storage in StorSimple is thinly provisioned.

StorSimple provides a broad set of data management tools that enable customers to use data management services in Microsoft Azure in ways that are familiar to them, including archive and backup storage.

Cloud snapshots are point-in-time copies of data that are stored in cool tiers in the cloud. All cloud snapshots are fully reduced (deduplicated and compressed) to minimize the amount of storage that is consumed.

A StorSimple solution transfers and stores data that already has been fully reduced. This minimizes the cost of cloud storage, the transaction costs, and the WAN bandwidth that are associated with storing data in the cloud.

In a StorSimple system, data that is ranked lowest is sent to a cool tier in the cloud, where it remains until it is accessed again and promoted to the warm tier.

StorSimple systems provide volume-level cloud mapping between storage volumes on the StorSimple system and the Microsoft Azure public cloud. Different volumes can have cool tiers in the same or different Microsoft Azure storage accounts. Every StorSimple system keeps a metadata map that describes the state of the system and provides an image of the volume’s contents at the time a snapshot is taken. This map is typically .1% the size of the stored data.

AES-256 encryption is applied to all data that is transmitted and stored in the cloud by the StorSimple solution to ensure its security. SHA-256 hashing is applied to all data that is transmitted and stored in the cloud as a means to guarantee data integrity.

Cloud clones are the equivalent of a synthetic full backup that have all of the current data for a volume at the time of the last snapshot. They are stored in the cool tier for use in disaster-recovery scenarios, but they occupy separate repositories from cloud snapshots, and they can reside within the same or a different cloud repository as the volume’s cloud snapshots. The following graphic shows cloud clones that are located in different repositories and shows that they can use the same or a different cloud service.

Figure 28 Distributed storage

A thin restore is a disaster-recovery process whereby a StorSimple system downloads data from the cloud. The first thing that is downloaded is the metadata map, after which users and applications can start accessing their working sets and download them. As data is downloaded, it is ranked and placed in the appropriate tier.

Figure 29 StorSimple backup

Thin restores tend to have extremely short recovery time objectives, because systems can begin accessing data after the metadata map has been downloaded. Thin restores do not restore cool data that does not belong to any working sets.

Location-independent recovery refers to the ability to perform thin restores from any location that has a suitable Internet connection. This differs from legacy disaster recovery operations, which are restricted to running at specific recovery sites.

Location independence adds an additional level of redundancy to the recovery process, and it does not require the capital investment of traditional replication solutions. A customer that has multiple data center locations can use StorSimple systems running in any of those locations to recover from disasters in any of the other sites. Similarly, a single StorSimple system can act as a spare for any of the others, providing an extremely cost-effective disaster recovery implementation.

11.5.2 Microsoft Azure and IaaS

There are a number of compute-related services in Microsoft Azure, including web and worker roles, cloud services, and HD Insight. The focus of this paper is hybrid cloud IaaS. Although some of the other compute capabilities will be covered briefly, the focus will be on virtual machines in Microsoft Azure.

11.5.2.1 Microsoft Azure Cloud Service

When you create a virtual machine or application and run it in Microsoft Azure, the virtual machine or the code and configuration together are called a Microsoft Azure cloud service.

By creating a cloud service, you can deploy multiple virtual machines or a multitier application in Microsoft Azure, define multiple roles to distribute processing, and allow flexible scaling of your application. A cloud service can consist of one or more virtual machines or web roles and worker roles, each of which has its own application files and configuration.

Virtual machines that are hosted in Microsoft Azure must be contained within cloud services. A single Microsoft Azure subscription by default is limited to 20 cloud services, and each cloud service can include up to 50 virtual machines.

11.5.2.2 Virtual Machines in Microsoft Azure

You can control and manage virtual machines in Microsoft Azure. After you create a virtual machine, you can delete and re-create it whenever you have to, and you can access the virtual machine just like any other server. You can use the following types of virtual hard disks (VHDs) to create a virtual machine:

· Image: A template that you use to create a new virtual machine. An image does not have specific settings (such as the computer name and user account settings) that a running virtual machine has. If you use an image to create a virtual machine, an operating system disk is created automatically for the new virtual machine.

· Disk: A VHD that you can start and mount as a running version of an operating system. After an image has been provisioned, it becomes a disk. A disk is always created when you use an image to create a virtual machine. Any VHD that is attached to virtualized hardware and running as part of a service is a disk.

You can use the following options to create a virtual machine from an image:

· Create a virtual machine by using a platform image from the Microsoft Azure Management Portal.

· Create and upload a .vhd file that contains an image to Microsoft Azure, and then use the uploaded image to create a virtual machine.

Microsoft Azure provides specific combinations of central processing unit (CPU) cores and memory for virtual machines. These combinations are known as virtual machine sizes. When you create a virtual machine, you select a specific size. This size can be changed after deployment. The sizes that are available for virtual machines are the following:

Virtual Machine Size	CPU Cores	Memory	Disk Space for Cloud Services	Disk Space for Virtual Machines	Maximum Data Disks (1 TB Each)	Maximum IOPS (500 Maximum per Disk)
ExtraSmall	Shared	768 MB	19 GB	20 GB	1	1 × 500
Small	1	1.75 GB	224 GB	70 GB	2	2 × 500
Medium	2	3.5 GB	489 GB	135 GB	4	4 × 500
Large	4	7 GB	999 GB	285 GB	8	8 × 500
ExtraLarge	8	14 GB	2,039 GB	605 GB	16	16 × 500
A5	2	14 GB			4	4 x 500
A6	4	28 GB	999 GB	285 GB	8	8 × 500
A7	8	56 GB	2,039 GB	605 GB	16	16 × 500

Source: Virtual Machine and Cloud Service Sizes for Microsoft Azure

Important Virtual machines will begin to incur cost as soon as they are provisioned, regardless of whether they are turned on.

11.5.2.3 Virtual Machine Storage in Microsoft Azure

A virtual machine in Microsoft Azure is created from an image or a disk. All virtual machines use one operating system disk, a temporary local disk, and possibly multiple data disks. All images and disks, except for the temporary local disk, are created from VHDs, which are .vhd files that are stored as page blobs in a storage account in Microsoft Azure. You can use platform images that are available in Microsoft Azure to create virtual machines, or you can upload your images to create customized virtual machines. The disks that are created from images are also stored in Microsoft Azure storage accounts. You can create new virtual machines easily by using existing disks.

VHD Files

A .vhd file is stored as a page blob in Microsoft Azure Storage, and it can be used for creating images, operating system disks, or data disks in Microsoft Azure. You can upload a .vhd file to Microsoft Azure and manage it just as you would any other page blob. The .vhd files can be copied or moved, and they can be deleted as long as a lease does not exist on the VHD.

A VHD can be in a fixed format or a dynamic format. Currently however, only the fixed format of .vhd files is supported in Microsoft Azure. The fixed format lays out the logical disk linearly within the file, so that disk offset X is stored at blob offset X. At the end of the blob, a small footer describes the properties of the VHD. Often, the fixed format wastes space because most disks contain large unused ranges. However, in Microsoft Azure, fixed .vhd files are stored in a sparse format, so that you receive the benefits of fixed and dynamic disks at the same time.

When you create a virtual machine from an image, a disk is created for the virtual machine, which is a copy of the original .vhd file. To protect against accidental deletion, a lease is created if you create an image, an operating system disk, or a data disk from a .vhd file.

Before you can delete the original .vhd file, you must first delete the disk or image to remove the lease. To delete a .vhd file that is being used by a virtual machine as an operating system disk, you must delete the virtual machine, delete the operating system disk, and then delete the original .vhd file. To delete a .vhd file that is used as a source for a data disk, you must detach the disk from the virtual machine, delete the disk, and then delete the .vhd file.

Images

An image is a .vhd file that you can use as a template to create a new virtual machine. An image is a template because it does not have specific settings (such as the computer name and user account settings) that a configured virtual machine does. You can use images from the Image Gallery to create virtual machines, or you can create your own images.

The Microsoft Azure Management Portal enables you to choose from several platform images to create a virtual machine. These images contain the Windows Server 2012 R2, Windows Server 2012 or Windows Server 2008 R2 operating system, and several distributions of the Linux operating system. A platform image can also contain applications, such as SQL Server.

To create a Windows Server image, you must run the Sysprep command on your development server to generalize and shut it down before you can upload the .vhd file that contains the operating system.

Disks

You use disks in different ways with a virtual machine in Microsoft Azure. An operating system disk is a VHD that you use to provide an operating system for a virtual machine. A data disk is a VHD that you attach to a virtual machine to store application data. You can create and delete disks whenever you want to.

You choose from among multiple ways to create disks, depending on the needs of your application. For example, a typical way to create an operating system disk is to use an image from the Image Gallery when you create a virtual machine. You can create a new data disk by attaching an empty disk to a virtual machine. You can also create a disk by using a .vhd file that has been uploaded or copied to a storage account in your subscription. You cannot use the portal to upload .vhd files. If you want to upload .vhd files you can use other tools that work with Microsoft Azure storage accounts—including the Microsoft Azure PowerShell cmdlets—to upload or copy the file.

Operating System Disk

Every virtual machine has one operating system disk. You can upload a VHD that can be used as an operating system disk, or you can create a virtual machine from an image, and a disk is created for you. An operating system disk is a VHD that you can start and mount as a running version of an operating system. Any VHD that is attached to virtualized hardware and running as part of a service is an operating system disk. The maximum size of an operating system disk can be 127 GB.

When an operating system disk is created in Microsoft Azure, three copies of the disk are created for high durability. Additionally, if you choose to use a disaster recovery plan that is geo-replication–based, your VHD is also replicated at a distance of more than 400 miles away. Operating system disks are registered as SATA drives and labeled as drive C.

Data Disk

A data disk is a VHD that can be attached to a running virtual machine to persistently store application data. You can upload and attach to the virtual machine a data disk that already contains data, or you can use the Microsoft Azure Management Portal to attach an empty disk to the virtual machine. The maximum size of a data disk is 1 TB, and you are limited in the number of disks that you can attach to a virtual machine, based on the size of the virtual machine.

Data disks are registered as SCSI drives, and you can make them available for use within the operating system by using the Disk Manager in Windows. The maximum number of data disks per virtual machine size was shown in the preceding table.

If multiple data disks are attached to a virtual machine, striping inside the virtual machine operating system can be utilized to create a single volume on the multiple attached disks (a volume of up to 16 TB that consists of a stripe of sixteen 1 TB data disks). As mentioned previously, use of operating system striping is not possible with geo-redundant data disks.

Temporary Local Disk

Each virtual machine that you create has a temporary local disk, which is labeled as drive D. This disk exists only on the physical host server on which the virtual machine is running; it is not stored in blobs in Microsoft Azure Storage. This disk is used by applications and processes that are running in the virtual machine for transient and temporary storage of data. It is used also to store page files for the operating system.

Note  Any data located on a temporary drive will not survive a host-machine failure or any other operation that requires moving the virtual machine to another piece of hardware.

Use of the letter D for the drive is by default. You can change the letter as follows:

1. Deploy the virtual machine normally, with or without the second data disk attached. (The data disk initially will be drive E, if it is a formatted volume.)
2. Move the page file from drive D to drive C.
3. Reboot the virtual machine.
4. Swap the drive letters on the current drives D and E.
5. Optionally, move the page file back to the resource drive (now drive E).

If the virtual machine is resized or moved to new hardware for service healing, the drive naming will stay in place. The data disk will stay at drive D, and the resource disk will always be the first available drive letter (which would be E, in this example).

Host Caching

The operating system disk and data disk has a host caching setting (sometimes called hosted cache mode) that enables improved performance under some circumstances. However, these settings can have a negative effect on performance in other circumstances, depending on the application. By default, host caching is OFF for Read and Write operations for data disks. Host caching is ON by default for Read and Write operations for operating system disks.

RDP and Remote Windows PowerShell

New virtual machines that are created through the Microsoft Azure Management Portal have Remote Desktop Protocol (RDP) and Windows PowerShell remoting available.

11.5.2.4 Virtual Machine Placement and Affinity Groups

You can group the services in your Microsoft Azure subscription that must work together to achieve optimal performance by using affinity groups. When you create an affinity group, it lets Microsoft Azure know to keep all of the services that belong to your affinity group running on the same data center cluster. For example, if you wanted to keep different virtual machines close together, you would specify the same affinity group for those virtual machines and associated storage. Then when you deploy those virtual machines, Microsoft Azure will locate them in a data center as close to each other as possible. This reduces latency and increases performance, while potentially lowering costs.

Affinity groups are defined at the subscription level, and the name of each affinity group has to be unique within the subscription. When you create a new resource, you can use an affinity group that you previously created or create a new one.

11.5.2.5 Endpoints and ACLs

When you create a virtual machine in Microsoft Azure, it is fully accessible from any of your other virtual machines that are within the virtual network to which it is connected. All protocols (such as TCP, UDP, and Internet Control Message Protocol [ICMP]) are supported within the local virtual network. Virtual machines on your virtual network are automatically given an internal IP address from the private range (RFC 1918) that you defined when you created the network.

To provide access to your virtual machines from outside of your virtual network, you use the external IP address and configure public endpoints. These endpoints are similar to firewall and port forwarding rules, and they can be configured in the Microsoft Azure Management Portal. By default, when they are created by using the Microsoft Azure Management Portal, ports for RDP and Windows PowerShell remoting are opened. These ports use random public-port addresses, which are mapped to the correct ports on the virtual machines. You can remove these preconfigured endpoints if you have network connectivity through a VPN.

A network access control list (ACL) is a security enhancement that is available for your Microsoft Azure deployment. An ACL provides the ability to selectively permit or deny traffic for a virtual machine endpoint. This packet filtering capability provides an additional layer of security. Currently, you can specify network ACLs for virtual machine endpoints only. You cannot specify an ACL for a virtual network or a specific subnet that is contained in a virtual network.

By using network ACLs, you can do the following:

· Selectively permit or deny incoming traffic based on a remote subnet IPv4 address range to a virtual machine input endpoint

· Ban IP addresses

· Create multiple rules per virtual machine endpoint

· Specify up to 50 ACL rules per virtual machine endpoint

· Organize the rules to ensure that the correct set of rules are applied on a given virtual machine endpoint (lowest to highest)

· Specify an ACL for a specific remote subnet IPv4 address

An ACL is an object that contains a list of rules. When you create an ACL and apply it to a virtual machine endpoint, packet filtering takes place on the host node of your virtual machine. This means the traffic from remote IP addresses is filtered by the host node to match ACL rules, instead of on your virtual machine. This prevents your virtual machine from spending precious CPU cycles on packet filtering.

When a virtual machine is created, a default ACL is put in place to block all incoming traffic. However, if an endpoint is created for Port 3389, the default ACL is modified to allow all inbound traffic for that endpoint. Inbound traffic from any remote subnet is then allowed to that endpoint and no firewall provisioning is required. All other ports are blocked for inbound traffic unless endpoints are created for those ports. Outbound traffic is allowed by default.

You can selectively permit or deny network traffic for a virtual machine input endpoint by creating rules that specify “permit” or “deny.” By default, when an endpoint is created, all traffic is denied to the endpoint. For that reason, it’s important to understand how to create Permit and Deny rules and place them in the proper order of precedence if you want granular control over the network traffic that you choose to allow to reach the virtual machine endpoint.

Points to consider:

• No ACL: By default when an endpoint is created, we permit all for the endpoint.
• Permit: When you add one or more “permit” ranges, you are denying all other ranges by default. Only packets from the permitted IP range can communicate with the virtual machine endpoint.
• Deny: When you add one or more Deny ranges, you are permitting all other ranges of traffic by default.
• Combination of Permit and Deny: You can use a combination of Permit and Deny when you want to set a specific IP range to be permitted or denied.

Network ACLs can be set up on specific virtual machine endpoints. For example, you can specify a network ACL for an RDP endpoint that is created on a virtual machine, which locks down access for certain IP addresses. The following table shows a way to grant access to public virtual IPs of a certain range to permit access for RDP. All other remote IPs are denied. It follows a “lowest takes precedence” rule order.

Network ACLs can be specified on a load-balanced set endpoint. If an ACL is specified for a load-balanced set, the network ACL is applied to all virtual machines in that set. For example, if a load-balanced set is created with Port 80 and the set contains three virtual machines, the network ACL created on endpoint Port 80 of one virtual machine will automatically apply to the other virtual machines.

Figure 30 Load balanced endpoints

11.5.2.6 Virtual Machine High Availability

You can ensure the availability of your application by using multiple virtual machines in Microsoft Azure. By using multiple virtual machines in your application, you can make sure that your application is available during local network failures, local disk-hardware failures, and any planned downtime that the platform might require.

You manage the availability of your application that uses multiple virtual machines by adding the virtual machines to an availability set. Availability sets are directly related to fault domains and update domains. A fault domain in Microsoft Azure is defined by avoiding single points of failure, like the network switch or power unit of a rack of servers. In fact, a fault domain is closely equivalent to a rack of physical servers. When multiple virtual machines are connected together in a cloud service, an availability set can be used to ensure that the virtual machines are located in different fault domains. The following diagram shows two availability sets, each of which contains two virtual machines.

Figure 31 Availability sets

Microsoft Azure periodically updates the operating system that hosts the instances of an application. A virtual machine is shut down when an update is applied. An update domain is used to ensure that not all of the virtual machine instances are updated at the same time.

When you assign multiple virtual machines to an availability set, Microsoft Azure ensures that the virtual machines are assigned to different update domains. The previous diagram shows two virtual machines running Internet Information Services (IIS) in separate update domains and two virtual machines running SQL Server also in separate update domains.

Important The high availability concepts for virtual machines in Microsoft Azure are not the same as on-premises Hyper-V. Microsoft Azure does not support live migration or movement of running virtual machines. For high availability, multiple virtual machines per application or role must be created. Microsoft Azure constructs such as availability groups, and load balancing must be utilized. Each application that is being considered for deployment must be analyzed to determine how the high availability features in Microsoft Azure can be utilized. If a given application cannot use multiple roles or instances for high availability (meaning that a single virtual machine that is running the application must be online at all times), Microsoft Azure cannot support that requirement.

You should use a combination of availability sets and load-balanced endpoints (discussed in subsequent sections) to help ensure that your application is always available and running efficiently.

For more information about host updates for Microsoft Azure and how they affect virtual machines and services, see Microsoft Azure Host Updates: Why, When, and How.

11.5.2.7 Virtual Machine Load Balancing

External communication with virtual machines can occur through Microsoft Azure endpoints. These endpoints are used for different purposes, such as load-balanced traffic or direct virtual machine connectivity like RDP or SSH. You define endpoints that are associated with specific ports and are assigned a specific communication protocol. An endpoint can be assigned to either a TCP or UDP based transport protocol (the TCP protocol includes HTTP and HTTPS traffic).

Each endpoint that is defined for a virtual machine is assigned a public and private port for communication. The private port is defined for setting up communication rules on the virtual machine, and the public port is used by Microsoft Azure to communicate with the virtual machine from external resources.

If you configure it, Microsoft Azure provides round-robin load balancing of network traffic to publicly defined ports of a cloud service. When your cloud service contains instances of web roles or worker roles, you enable load balancing by setting the number of instances that are running in the service to greater-than or equal-to two and by defining a public endpoint in the service definition. For virtual machines, you can set up load balancing by creating new virtual machines, connecting them under a cloud service, and adding load-balanced endpoints to the virtual machines.

A load-balanced endpoint is a specific TCP or UDP endpoint that is used by all virtual machines that are contained in a cloud service. The following image shows a load-balanced endpoint that is shared among three virtual machines and uses a public and private Port 80:

Figure 32 More load balanced endpoints

A virtual machine must be in a healthy state to receive network traffic. You can optionally define your own method for determining the health of the virtual machine by adding a load-balancing probe to the load-balanced endpoint. Microsoft Azure probes for a response from the virtual machine every 15 seconds and takes a virtual machine out of the rotation if no response has been received after two probes. You must use Windows PowerShell to define probes on the load balancer.

11.5.2.8 Limitations of Virtual Machines in Microsoft Azure

Although virtual machines in Microsoft Azure are full running instances of Windows or Linux from an operating system perspective, in some cases (because they are virtual machines or are running on a cloud infrastructure), some operating system features and capabilities might not be supported.

The following table provides examples of Windows operating system features that are not supported for virtual machines in Microsoft Azure:

Operating System Roles or Feature	Explanation
Hyper-V	It is not supported to run Hyper-V within a virtual machine that is already running on Hyper-V.
Dynamic Host Configuration Protocol (DHCP)	Virtual machines in Microsoft Azure do not support broadcast traffic to other virtual machines.
Failover clustering	Microsoft Azure does not handle failover clustering’s “virtual” or floating IP addresses for network resources.
BitLocker on operating system disk	Microsoft Azure does not support the Trusted Platform Module (TPM).
Client operating systems	Licenses for Microsoft Azure do not support client operating systems.
Virtual Desktop Infrastructure (VDI) using Remote Desktop Services (RDS)	Licenses for Microsoft Azure do not support running VDI virtual machines through RDS.

More Microsoft applications are being tested and supported for deployment of virtual machines in Microsoft Azure. For more information, see Microsoft Server Software Support for Microsoft Azure Virtual Machines.

11.5.3 Networking in Microsoft Azure

Network services in Microsoft Azure provide a variety of solutions for network connectivity within Microsoft Azure, and between the on-premises infrastructure and Microsoft Azure. In 2012, Microsoft Azure made substantial upgrades to the fabric and network architecture to flatten the design and significantly increase the horizontal (or node-to-node) bandwidth that is available.

With software improvements, these upgrades provide significant bandwidth between compute and storage that uses a flat-network topology. The specific implementation of the flat network for Microsoft Azure is referred to as the “Quantum 10” (Q10) network architecture. Q10 provides a fully non-blocking, 10 Gbps–based, fully meshed network. It provides an aggregate backplane in excess of 50 terabytes per second (Tbps) of bandwidth for each data center running Microsoft Azure.

Another major improvement for reliability and throughput is moving from a hardware load balancer to a software load balancer. After these upgrades, the storage architecture and design that were described in previous sections was tuned to leverage the new Q10 network fully to provide flat-network storage for Microsoft Azure Storage.

For architectural details of the Q10 design, see VL2: A Scalable and Flexible Data Center Network.

11.5.3.1 Virtual Network in Microsoft Azure

Virtual Network in Microsoft Azure enables you to create secure site-to-site connectivity and protected private virtual networks in the cloud. You can specify the address space that will be used for your virtual network and the virtual network gateway. Additionally, new name-resolution features allow you to connect directly to role instances and virtual machines by host name. These features allow you to use Microsoft Azure as a protected private virtual network in the cloud.

Before you configure Virtual Network, you should carefully consider possible scenarios. For this release, it can be difficult to make changes after your virtual network has been created and you have deployed role instances and virtual machines. After this stage of deployment, you cannot easily modify the baseline network configuration, and many values cannot be modified without pulling back roles and virtual machines, and then reconfiguring them. Therefore, you should not attempt to create a virtual network and then try to adapt the scenario to fit the network. Scenarios that are enabled by Virtual Network in Microsoft Azure include:

· Create secure site-to-site network connectivity between Microsoft Azure and your on-premises network, effectively creating a virtual branch office or data center in the cloud. This is possible by using a hosted VPN gateway and a supported VPN gateway device (including Routing and Remote Access service (RRAS) in Windows Server 2012 R2 and Windows Server 2012).

· Extend your enterprise networks to Microsoft Azure.

· Migrate existing applications and services to Microsoft Azure.

· Host name resolution. You can specify an on-premises Domain Name System (DNS) server or a dedicated DNS server that is running elsewhere.

· Persistent dynamic IP addresses for virtual machines. This means that the internal IP address of your virtual machines will not change, even when you restart a virtual machine.

· Join virtual machines that are running in Microsoft Azure to your domain that is running on-premises.

· Create point-to-site virtual networks, which enables individual workstations to establish VPN connectivity to Microsoft Azure virtual networks—for example, for developers at a remote site to connect to network services in Microsoft Azure.

Virtual Network in Microsoft Azure has the following properties:

· Virtual machines can have only one IP address (or one IP address plus a virtual IP address, if they are load balanced; note that the virtual IP address is not assigned to the virtual machine itself, it’s assigned to the gateway device that forwards packets to the virtual machine).

· Every virtual machine gets an IP address from DHCP; static IP addresses are not supported.

· Virtual machines on the same virtual network can communicate.

· Virtual machines on different virtual networks cannot communicate directly and must loop back through the Internet to communicate.

· Egress traffic from Microsoft Azure is charged.

· Ingress traffic to Microsoft Azure is free (not charged).

· All virtual machines by default have Internet access. There is currently no official way to force Internet traffic to go through on-premises devices, such as proxies.

· There is only one virtual gateway per virtual network.

As mentioned previously, virtual networks and subnets in Microsoft Azure must utilize private (RFC 1918) IP-address ranges.

11.5.3.2 Site-to-Site VPN in Microsoft Azure

You can link Virtual Network in Microsoft Azure to an on-premises network through a site-to-site VPN connection.

To create a secure VPN connection, the person who will configure the VPN device must coordinate with the person who will create the Management Portal configuration. This coordination is required, because the Management Portal requires IP address information from the VPN device to start the VPN connection and create the shared key. The shared key is then exported to configure the VPN gateway device and complete the connection.

Sample configuration scripts are available for many, but not all, VPN devices. If your VPN device is in the list of supported devices, you can download the corresponding sample configuration script to help you configure the device. If you do not see your VPN device in the list, your device still might work with Virtual Network in Microsoft Azure if it satisfies the requirements. For more information, see About VPN Devices for Virtual Network.

11.5.3.3 Point-to-Site VPN in Microsoft Azure

The point-to-site VPN in Microsoft Azure allows you to set up VPN connections between individual computers and Virtual Network in Microsoft Azure without a VPN device on-premises. This feature is called Point-to-Site Virtual Private Networking, which is the same as a remote access VPN connection from the client to the Azure Virtual Network. It greatly simplifies the setup of secure connections between Microsoft Azure and client computers, whether from an office environment or from remote locations.

It is especially useful for developers who want to connect to Virtual Network in Microsoft Azure (and to the individual virtual machines within it) from behind a corporate firewall or from a remote location.

Because the connection is point-to-site, they do not need their IT staff to perform any activities to enable it, and no VPN hardware needs to be installed or configured. Instead, you can simply use the VPN client that is built-in to Windows to tunnel to Virtual Network in Microsoft Azure. This tunnel uses the Secure Sockets Tunneling Protocol (SSTP), and it can traverse firewalls and proxies automatically, while giving you complete security.

Here’s a visual representation of the point-to-site scenarios:

Figure 33: Connecting virtual networks

11.5.3.4 Affinity Groups

After you have created a virtual network, an affinity group is also created. When you create resources (such as storage accounts) in Microsoft Azure, an affinity group tells Window Azure that you want to keep these resources located together. When you have an affinity group, you should reference this always when you are creating related resources.

11.5.3.5 Name Resolution

Name resolution is an important consideration for virtual network design. Even though you may create a secure site-to-site VPN connection, communication by host name is not possible without name resolution.

To refer to virtual machines and role instances within a cloud service by host name directly, Microsoft Azure provides a name resolution service. This service is used for internal host name resolution within a cloud service. The name resolution service that is provided by Microsoft Azure is a completely separate service from that which is used to access your public endpoints on the Internet.

Before you deploy role instances or virtual machines, you must consider how you want name resolution to be handled. Two options are available: you can use the internal name resolution that is provided by Microsoft Azure or choose to specify a DNS server that is not maintained by Microsoft Azure. Not all configuration options are available for every deployment type. Carefully consider your deployment scenario before you make this choice.

11.5.3.6 DNS Considerations

When you define a virtual network, Microsoft Azure will provide a DNS Server service. However, if you want to use your existing DNS infrastructure for name resolution, or you have a dependency on Active Directory, you need to define your own. Defining your own in the virtual network configuration doesn’t actually create a DNS server; instead you are configuring the DHCP Server service to include the DNS server IP that you define. This DNS server could be a reference to an existing on-premises DNS server, or a new DNS server that you will provision in the cloud.

Configuring your virtual network to use the name resolution service that is provided by Microsoft Azure is a relatively simple option. However, you may require a more full-featured DNS solution to support virtual machines or complex configurations. Your choice of name resolution method should be based on the scenario that it will support. You can use the following table to help make your decision.

Scenario	Name Resolution	Points to Consider
Cross-premises: Name resolution between role instances or virtual machines in Microsoft Azure and on-premises computers	DNS solution of your choice (not Microsoft Azure service)	· Name resolution (DNS) design · Address space · Supported VPN gateway device · Internet-accessible IP address for your VPN gateway device
Cross-premises: Name resolution between on-premises computers and role instances or virtual machines in Microsoft Azure	DNS solution of your choice (not Microsoft Azure service)	· Name resolution (DNS) design · Address space · Supported VPN gateway device · Internet-accessible IP address for your VPN gateway device
Name resolution between role instances located in the same cloud service	Microsoft Azure name resolution service (internal)	· Name resolution (DNS) design
Name resolution between virtual machines located in the same cloud service	Microsoft Azure name resolution service (internal)	· Name resolution (DNS) design
Name resolution between virtual machines and role instances located in the same Virtual Network, but different cloud services	DNS solution of your choice (or Microsoft Azure service)	· Name resolution (DNS) design · Address space · Supported VPN gateway device · Internet-accessible IP address for your VPN gateway device
Name resolution between virtual machines and role instances that are located in the same cloud services but not in a Microsoft Azure Virtual Network.	Not applicable	· Virtual machines and role instances cannot be deployed in the same cloud service
Name resolution between role instances that are located in different cloud services but not in a Microsoft Azure Virtual Network.	Not applicable	· Connectivity between virtual machines or role instances in different cloud services is not supported outside a virtual network
Name resolution between virtual machines that are located in the same Microsoft Azure Virtual Network.	DNS solution of your choice (not Microsoft Azure service)	· Name-resolution (DNS) design · Address space · Supported VPN gateway device · Internet-accessible IP address for your VPN gateway device
Use name resolution to direct traffic between datacenters.		See Microsoft Azure Traffic Manager
Control the distribution of user traffic to services hosted by Microsoft Azure hosted.		See Microsoft Azure Traffic Manager

Although the name resolution service that is provided by Microsoft Azure requires very little configuration, it is not the appropriate choice for all deployments. If your network requires name resolution across cloud services or across premises, you must use your own DNS server. If you want to register additional DNS records, you will have to use a DNS solution that is not provided by Microsoft Azure.

11.5.3.7 Microsoft Azure Traffic Manager

The Microsoft Azure Traffic Manager allows you to control the distribution of user traffic to Microsoft Azure hosted services. The services can be running in the same data center or in different centers around the world. Traffic Manager works by applying an intelligent policy engine to the DNS queries in your domain names.

The following conceptual diagram demonstrates Traffic Manager routing. The user uses the www.contoso.com company domain and to reach a hosted service to accept the request. The Traffic Manager policy dictates which hosted service receives the request. Although Traffic Manager conceptually routes traffic to a given hosted service, the actual process is slightly different because it uses DNS. No actual service traffic routes through Traffic Manager. The client computer calls the hosted service directly when Traffic Manager resolves the DNS entry for the company domain to the IP address of a hosted service.

Figure 34: Name resolution

The numbers in the preceding diagram correspond to the numbered descriptions in the following list:

1. User traffic to company domain: The user requests information by using the company domain name. The typical process to resolve a DNS name to an IP address begins. Company domains must be reserved through normal Internet domain-name registration processes and are maintained outside of Traffic Manager. In this diagram, the company domain is www.contoso.com.
2. Company domain to Traffic Manager domain: The DNS resource record for the company domain points to a Traffic Manager domain that is maintained in Microsoft Azure Traffic Manager. In the example, the Traffic Manager domain is contoso.trafficmanager.net.
3. Traffic Manager domain and policy: The Traffic Manager domain is part of the Traffic Manager policy. Traffic enters through the domain. The policy dictates how to route that traffic.
4. Traffic Manager policy rules processed: The Traffic Manager policy uses the chosen load-balancing method and monitoring status to determine which Microsoft Azure hosted service should service the request.
5. Hosted-service domain name sent to user: Traffic Manager returns the DNS name of the hosted service to the IP address of a chosen hosted service to the user. The user’s local DNS server resolves the domain to the IP address of the chosen hosted service.
6. User calls hosted service: The client computer calls the chosen hosted service directly by using the returned IP address. Because the company domain and resolved IP address are cached on the client computer, the user continues to interact with the chosen hosted service until its local DNS cache expires.
   Note  The Windows operating system caches DNS host entries for the duration of their Time to Live (TTL). Whenever you evaluate Traffic Manager policies, retrieving host entries from the cache bypasses the policy, and you can observe unexpected behavior. If the TTL of a DNS host entry in the cache expires, new requests for the same host name should result in the client running a fresh DNS query. However, browsers typically cache these entries for longer periods, even after their TTL has expired. To reflect the behavior of a Traffic Manager policy accurately when accessing the application through a browser, it is necessary to force the browser to clear its DNS cache before each request.
7. Repeat: The process repeats itself when the client’s DNS cache expires. The user might receive the IP address of a different hosted service, depending on the load-balancing method that is applied to the policy and the health of the hosted service at the time of the request.

The following links include additional details about this process:

· About Traffic Manager Load Balancing Methods

· About Traffic Manager Monitoring

· Traffic Manager Overview

· Operations for Traffic Manager

11.5.3.8 Microsoft Azure Content Delivery Network

Microsoft Azure Content Delivery Network (CDN) offers developers a global solution for delivering high-bandwidth content by caching blobs and static content of compute instances on physical nodes in the United States, Europe, Asia, Australia, and South America. For a current list of CDN node locations, see Microsoft Azure CDN Node Locations.

The benefits of using CDN to cache Microsoft Azure data include:

· Better performance and user experience for users who are far from a content source and are using applications for which many “Internet trips” are required to load content.

· Large distributed scale to better handle instantaneous high load—for example, at the start of an event, such as a product launch.

To use Microsoft Azure CDN, you must have a Microsoft Azure subscription and enable the feature on the storage account or hosted service in the Microsoft Azure Management Portal. CDN is an add-on feature to your subscription, and it has a separate billing plan. For more information, see Microsoft Azure Pricing.

12 Fabric and Fabric Management

The PLA patterns at a high level include the concepts of compute, storage, and network fabrics. This is logically and physically independent from components, such as the components in System Center 2012 R2 that provides management of the underlying fabric.

Figure 33: Fabric and fabric management components

12.1 Fabric

The definition of the fabric is all of the physical and virtual resources under the scope of management of the fabric management infrastructure. The fabric is typically the entire compute, storage, and network infrastructure—usually implemented as Hyper-V host clusters—being managed by the System Center infrastructure.

For private cloud infrastructures, the fabric constitutes a resource pool that consists of one of more scale units. In a modular architecture, the concept of a scale unit refers to the point to which a module in the architecture can scale before another module is required. For example, an individual server is a scale unit, because it can be expanded to a certain point in terms of CPU and RAM; however, once it reaches its maximum scalability, an additional server is required to continue scaling. Each scale unit also has an associated amount of physical installation and configuration labor. With large scale units, like a preconfigured full rack of servers, the labor overhead can be minimized.

It is critical to know the scale limits of all hardware and software components when determining the optimum scale units for the overall architecture. Scale units enable the documentation of all the requirements (for example, space, power, HVAC, or connectivity) needed for implementation.

12.2 Fabric Management

Fabric management is the concept of treating discrete capacity pools of servers, storage, and networks as a single fabric. The fabric is then subdivided into capacity clouds, or resource pools, which carry characteristics like delegation of access and administration, service-level agreements (SLAs), and cost metering. Fabric management enables the centralization and automation of complex management functions that can be carried out in a highly standardized, repeatable fashion to increase availability and lower operational costs.

12.2.1 Fabric Management Host Architecture

In a private cloud infrastructure, it is recommended that the systems that make up the fabric resource pools be physically separate from the systems that provide fabric management. Much like the concept of having a top-of-rack (ToR) switch, it is recommended to provide separate fabric management hosts to manage the underlying services that provide capacity to the private cloud infrastructure. This model helps make sure that the availability of the fabric is separated from fabric management, and regardless of the state of the underlying fabric resource pools, management of the infrastructure and its workloads is maintained at all times.

To support this level of availability and separation, private cloud architectures should contain a separate set of hosts (minimum of two) configured as a failover cluster in which the Hyper-V role is enabled. Furthermore, these hosts should contain highly availability virtualized instances of the management infrastructure (System Center) to support fabric management operations that are stored on dedicated CSVs.

13 Non-Converged Architecture Pattern

This section contains an architectural example that is based on the non-converged pattern validation requirements that were outlined in the previous sections. This example provides guidance about the hardware that is required to build the non-converged pattern reference architecture by using high-level, non-OEM–specific system models.

As explained earlier, the non-converged pattern comprises traditional blade or non-blade servers that utilize a standard network and storage network infrastructure to support a high availability Hyper-V failover cluster fabric infrastructure. This infrastructure pattern provides the performance of a large-scale Hyper-V host infrastructure and the flexibility of utilizing existing infrastructure investments at a lower cost than a converged architecture.

Figure 25 outlines a logical structure of components that follow this architectural pattern.

Figure 34 Non-converged architecture pattern

13.1 Compute

The compute infrastructure is one of the primary elements that must scale to support a large number of workloads. In a non-converged fabric infrastructure, a set of hosts that have the Hyper-V role enabled provide the fabric with the capability to achieve scale in the form of a large-scale failover cluster.

Figure 26 provides an overview of the compute layer of the private cloud fabric infrastructure.

Figure 35 Compute minimum configuration

13.1.1 Hyper-V Host Infrastructure

The Hyper-V host infrastructure is comprised of a minimum of four hosts and a maximum of 64 hosts in a single Hyper-V failover-cluster instance. Although Windows Server 2012 R2 failover clustering supports a minimum of two nodes, a configuration at that scale does not provide a sufficient reserve capacity to achieve cloud attributes such as elasticity and resource pooling.

As with any failover-cluster configuration, reserve capacity must be accounted for in the host infrastructure. Adopting a simple n-1 methodology does not always provide a sufficient amount of reserve capacity to support the workloads that are running on the fabric infrastructure. For true resilience to outages, we recommend that you size the reserve capacity within a single scale unit to one or more hosts. This is critical for delivering availability within a private cloud infrastructure and it is a key consideration when you are advertising the potential workload capacity of the fabric infrastructure.

Equally important to the overall density of the fabric is the amount of physical memory that is available for each fabric host. For service provider and enterprise configurations, a minimum of 192 GB of memory is required. As the demand for memory within workloads increases, this becomes the second largest factor for scale and density in the compute fabric architecture.

As discussed earlier, Hyper-V provides Dynamic Memory to support higher densities of workloads through a planned oversubscription model. Although it is safe to assume that this feature will provide increased density for the fabric, a private cloud infrastructure should carefully consider the use of Hyper-V Dynamic Memory as part of the compute design due to supportability limitations and performance requirements in certain workloads. Always refer to the vendor workload recommendations and support guidelines when you enable Hyper-V Dynamic Memory.

Additional considerations that should be accounted for in density calculations include:

· The amount of startup RAM that is required for each operating system

· The minimum RAM that is allocated to the virtual machine after startup for normal operations

· The maximum RAM that is assigned to the system to prevent oversubscription scenarios when memory demand is high

The Hyper-V parent partition (host) must have sufficient memory to provide services such as I/O virtualization, snapshot, and management to support the child partitions (guests). Previous guidance was provided to tune the parent partition reserve; however; when Dynamic Memory is used, root reserve is calculated automatically (based on the root physical memory and NUMA architecture of the hosts) and no longer requires manual configuration.

Although guidance about network connectivity that uses onboard network connections is provided in the following section, you should make sure that out-of-band (OOB) network-management connectivity is provided to support the remote management and provisioning capabilities that are found within System Center.

To address these capabilities, the compute infrastructure should support a minimum of one OOB management interface, with support for Intelligent Platform Management Interface (IPMI) 1.5/Data Center Management Interface (DCMI) 1.0 or Systems Management Architecture for Server Hardware (SMASH) 1.0 over WS-Man. Failure to include this component will result in a compute infrastructure that cannot utilize automated provisioning and management capabilities in the private cloud solution.

It should be assumed that customers will also require multiple types (or classifications) of resource pools to support a number of scenarios and associated workloads. These types of resource pools are expected to be evaluated as part of the capabilities that the resulting fabric will be required to provide.

For example, a resource pool that is intended for VDI resources might have different hardware, such as specialized graphics cards, to support RemoteFX capabilities within Hyper-V. For these reasons, options for a compute infrastructure that provide advanced resource pool capabilities, such as the RemoteFX resource pool, should be available to address these needs and provide a complete solution.

13.2 Network

When you are designing the fabric network for the Hyper-V failover cluster in Windows Server 2012 R2, it is important to provide the necessary hardware and network throughput to provide resiliency and Quality of Service (QoS). Resiliency can be achieved through availability mechanisms, and QoS can be provided through dedicated network interfaces or through a combination of hardware and software QoS capabilities.

Figure 27 provides an overview of the network layer of the private cloud fabric infrastructure.

Figure 36 Network minimum configuration

13.2.1 Host Connectivity

During the design of the network topology and associated network components of the private cloud infrastructure, the following key considerations apply:

· Provide adequate network port density: Designs should contain top-of-rack switches with sufficient density to support all host network interfaces.

· Provide adequate interfaces to support network resiliency: Designs should contain a sufficient number of network interfaces to establish redundancy through NIC Teaming.

· Provide network Quality of Service (QoS): Although dedicated cluster networks is an acceptable way to achieve QoS, utilizing high-speed network connections in combination with hardware- or software-defined network QoS policies provides a more flexible solution.

For PLA pattern designs, a minimum of two 10 GbE network interfaces and one OOB management connection is assumed a minimum baseline of network connectivity for the fabric architecture. Two interfaces are used for cluster traffic, and the third is available as a management interface. To provide resiliency, additional interfaces can be added and teamed by using the NIC Teaming feature in Windows Server 2012 R2.

It is recommended to have redundant network communication between all private cloud cluster nodes. As previously described, host connectivity in a private cloud infrastructure should support the following types of communication that are required by Hyper-V and the failover clusters that make up the fabric:

· Host management

· Virtual machine

· Live migration

· iSCSI (if required)

· Intra-cluster communication and CSV

Host management consists of isolated network traffic to manage the parent partition (host), and virtual machine traffic is on an accessible network for clients to access the virtual machines. The usage of the virtual machine traffic is highly dependent on the running workload and the interaction of the client with that application or service.

Live migration traffic is intermittent and used during virtual machine mobility scenarios such as planned failover events. This has the potential to generate a large amount of network traffic over short periods during transition between nodes. Live migration will default to the second lowest metric if three or more networks are configured in failover clustering.

When iSCSI is used, a dedicated storage network should be deployed within the fabric (because this is the non-converged pattern where storage traffic has a dedicated network). These interfaces should be disabled for cluster use, because cluster traffic can contribute to storage latency. Intracluster communication and CSV traffic consist of the following traffic types:

· Network health monitoring

· Intra-cluster communication

· CSV I/O redirection

Network health monitoring traffic consists of heartbeats that are sent to monitor the health status of network interfaces in a full mesh manner. This lightweight unicast traffic (approximately 134 bytes) is sent between cluster nodes over all cluster-enabled networks.

Because of its sensitivity to latency, bandwidth is important, as opposed to Quality of Service, because if heartbeat traffic becomes blocked due to network saturation, fabric nodes could be removed from cluster membership. By default, nodes exchange these heartbeats every one second, and they are considered to be down if they do not respond to five heartbeats.

Intra-cluster communication is variable (based on workload), and it is responsible for sending database updates and synchronizing state changes between the nodes in the fabric cluster. This lightweight traffic communicates over a single interface. As with network health monitoring, bandwidth is the primary concern, because this type of traffic is sensitive to latency during state changes, such as failover.

CSV I/O redirection traffic consists of lightweight metadata updates, and it can communicate over the same interface as the intracluster communication mentioned previously. It requires a defined Quality of Service to function properly. CSV uses SMB to route I/O over the network between nodes during failover events, so sufficient bandwidth is required to handle the forwarded I/O between cluster nodes. Additionally, CSV traffic will utilize SMB Multichannel and advanced network adapter capabilities such as RDMA; however, use of Jumbo Frames has shown little increase in performance.

13.3 Storage

Storage provides the final component for workload scaling, and as for any workload, storage must be designed properly to provide the required performance and capacity for overall fabric scale. In a non-converged fabric infrastructure, traditional SAN infrastructures that are connected over Fibre Channel or iSCSI provide the fabric with sufficient capacity to achieve storage scale.

Figure 28 provides an overview of the storage infrastructure for the non-converged pattern.

Figure 37 Storage minimum configuration

13.3.1 Storage Connectivity

For the operating system volume of the parent partition that is using direct-attached storage to the host, an internal SATA or SAS controller is required, unless the design utilizes SAN for all system storage requirements, including boot from SAN for the host operating system (Fibre Channel and iSCSI boot are supported in Windows Server 2012 R2). Depending on the storage protocol and devices that are used in the non-converged storage design, the following adapters are required to allow shared storage access:

· If using Fibre Channel SAN, two or more host bus adapters (HBAs)

· If using iSCSI, two or more 10 GbE network adapters or HBAs

As described earlier, Hyper-V in Windows Server 2012 R2 supports the ability to present SAN storage to the guest workloads that are hosted on the fabric infrastructure by using virtual Fibre Channel adapters. Virtual SANs are logical equivalents of virtual network switches within Hyper-V, and each virtual SAN maps to a single physical Fibre Channel uplink. To support multiple HBAs, a separate virtual SAN must be created per physical Fibre Channel HBA and mapped exactly to its corresponding physical topology.

When configurations use multiple HBAs, MPIO must be enabled within the virtual machine workload. A virtual SAN assignment should follow a pattern that is similar to a Hyper-V virtual switch assignment, in that if there are different classifications of service within the SAN, it should be reflected within the fabric.

As discussed in earlier sections, all physical Fibre Channel equipment must support NPIV. Hardware vendors must also provide drivers that display the Designed for Windows logo for all Fibre Channel HBAs, unless the drivers are provided in Windows Server 2012 R2. If zoning that is based on physical Fibre Channel switch ports is part of the fabric design, all physical ports must be added to allow for virtual machine mobility scenarios across hosts in the fabric cluster. Although virtual machines can support iSCSI boot, boot from SAN is not supported over the virtual Fibre Channel adapter and should not be considered as part of workload design.

13.3.2 Storage Infrastructure

The key attribute of the storage infrastructure for the non-converged pattern is the use of a traditional SAN infrastructure to provide access to storage to the fabric, fabric management, and workload layers. As discussed earlier, the primary reasons to adopt or maintain this design are to preserve existing investments in SAN or to maintain the current level of flexibility and capabilities that a SAN-based storage-array architecture provides.

For Hyper-V failover cluster and workload operations in a non-converged infrastructure, the fabric components utilize the following types of storage:

· Operating system: Non-shared physical boot disks (direct-attached storage or SAN) for the fabric management host servers

· Cluster witness: Shared witness disk or file share to support the failover cluster quorum

· Cluster Shared Volumes (CSV): One or more CSV LUNs for virtual machines (Fibre Channel or iSCSI), as presented by the SAN

· Guest clustering [optional]: Shared Fibre Channel, shared VHDX, or shared iSCSI LUNs for guest clustering

Figure 29 provides a conceptual view of the storage architecture for the non-converged pattern.

Figure 38 Non-converged architecture pattern

As outlined in the overview, fabric and fabric management host controllers require sufficient storage to account for the operating system and paging files. We recommend that virtual memory in Windows Server 2012 R2 be configured as “Automatically manage paging file size for all drives.”

Although boot from SAN by using Fibre Channel or iSCSI storage is supported in Windows Server 2012 R2, it is widely accepted to have storage configured locally per server to provide these capabilities for each server given the configuration of standard non-converged servers. In these cases, local storage should include two disks that are configured as RAID 1 (mirror) as a minimum, with an optional global hot spare.

To provide quorum for the server infrastructure, we recommend utilizing a quorum configuration of Node and Disk Majority. A cluster witness disk is required to support this quorum model. In non-converged pattern configurations, we recommend that you provide a 1 GB witness disk that is formatted as NTFS for all fabric and fabric management clusters. This provides resiliency and prevents partition-in-time scenarios within the cluster.

As described in earlier sections, Windows Server 2012 R2 provides multiple hosts access to a shared disk infrastructure through CSV. For non-converged patterns, the SAN should be configured to provide adequate storage for virtual machine workloads. Given that workload, virtual disks often exceed multiple gigabytes, so we recommend that where supported by the workload, dynamically expanding disks be used to provide higher density and more efficient use of storage. Additional SAN capabilities, such as thin provisioning of LUNs, can assist with the consumption of physical space. However, this functionality should be evaluated to help make sure that workload performance is not negatively affected.

For the purposes of Hyper-V failover clustering, CSV must be configured in Windows as a basic disk that is formatted as NTFS (FAT and FAT32 are not supported for CSV). CSV cannot be used as a witness disk, and they cannot have Windows Data Deduplication enabled. Although supported, ReFS should not be used in conjunction with a CSV with Hyper-V workloads. A CSV has no restrictions for the number of virtual machines that it can support on an individual CSV because metadata updates on a CSV are orchestrated on the server side, and they run in parallel to provide no interruption and increased scalability.

Performance considerations fall primarily on the IOPS that SAN provides, given that multiple servers from the Hyper-V failover cluster stream I/O to a commonly shared LUN. Providing more than one CSV to the Hyper-V failover cluster within the fabric can increase performance, depending on the SAN configuration.

To support guest clustering, LUNs can be presented to the guest operating system through iSCSI or Fibre Channel. Configurations for the non-converged pattern should include sufficient space on the SAN to support the number of LUNs that are needed for workloads with high-availability requirements that must be satisfied within the guest virtual machines and associated applications.

14 Converged Architecture Pattern

This section contains an architectural example that is based on the converged-pattern validation requirements that were previously outlined. This example will provide guidance about the hardware that is required to build the converged pattern reference architecture by using high-level, non-OEM–specific system models.

As explained earlier, the converged pattern comprises advanced blade servers that utilize a converged-network and storage-network infrastructure (often referred to as converged network architecture) to support a high availability Hyper-V failover-cluster fabric infrastructure. This infrastructure pattern provides the performance of a large-scale Hyper-V host infrastructure and the flexibility of utilizing software-defined networking capabilities at a higher system density than can be achieved through traditional non-converged architectures.

Although many aspects of converged architectures are the same, this section outlines the key differences between these two patterns. The following diagram outlines an example logical structure of components that follow the converged architectural pattern.

Figure 39 Converged architecture pattern

In the converged pattern, the physical converged network adapters (CNAs) are teamed, and they present network adapters and Fibre Channel HBAs to the parent operating system. From the perspective of the parent operating system, it appears that network adapters and Fibre Channel HBAs are installed. The configuration of teaming and other settings are performed at the hardware level.

14.1 Compute

As identified in the non-converged pattern, compute infrastructure remains as the primary element that provides fabric scale to support a large number of workloads. Identical to the non-converged pattern, the converged fabric infrastructure consists of an array of hosts that have the Hyper-V role enabled to provide the fabric with the capability to achieve scale in the form of a large-scale failover cluster.

Figure 31 provides an overview of the compute layer of the private cloud fabric infrastructure.

Figure 40 Compute minimum configuration

With the exception of storage connectivity, the compute infrastructure of the converged pattern is similar to the infrastructure of the non-converged pattern, because the Hyper-V host clusters utilize FCoE or iSCSI to connect to storage over a high-speed, converged network architecture.

14.1.1 Hyper-V Host Infrastructure

As in non-converged infrastructures, the server infrastructure comprises a minimum of four hosts and a maximum of 64 hosts in a single Hyper-V failover-cluster instance. Although Windows Server 2012 R2 failover clustering supports a minimum of two nodes, a configuration at that scale does not provide sufficient reserve capacity to achieve cloud attributes such as elasticity and resource pooling.

Converged infrastructures typically utilize blade servers and enclosures to provide compute capacity. In large-scale deployments in which multiple resource pools exist across multiple blade enclosures, a guideline of containing no more than 25 percent of a single cluster in a blade enclosure is recommended.

14.2 Network

When you are designing the fabric network for the Windows Server 2012 R2 Hyper-V failover cluster, it is important to provide the necessary hardware and network throughput to provide resiliency and Quality of Service (QoS). Resiliency can be achieved through availability mechanisms, and QoS can be provided through dedicated network interfaces or through a combination of hardware and software QoS capabilities.

Figure 32 provides an overview of the network layer of the private cloud fabric infrastructure.

Figure 41 Network minimum configuration

14.2.1 Host Connectivity

During the design of the network topology and associated network components of the private cloud infrastructure, the following key considerations apply:

· Provide adequate network port density—Designs should contain top-of-rack switches with sufficient density to support all host network interfaces.

· Provide adequate interfaces to support network resiliency—Designs should contain a sufficient number of network interfaces to establish redundancy through NIC teaming.

· Provide network Quality of Service— Having dedicated cluster networks is an acceptable way to achieve QoS, however the use of high-speed network connections in combination with either hardware-defined or software-defined network QoS policies provide a more flexible solution.

For PLA pattern designs, a minimum of two 10 GbE converged network adapters (CNAs) and one OOB management connection is a minimum baseline of network connectivity for the fabric architecture. Two interfaces are used for cluster traffic, and the third is available as a management interface. To provide resiliency, additional interfaces can be added and teamed by using a network adapter teaming solution from your OEM hardware. We recommend that you provide redundant network communication between all private cluster nodes. As previously described, host connectivity in a private cloud infrastructure should support the following types of communication that are required by Hyper-V and the failover clusters that make up the fabric, including:

· Host management

· Virtual machine

· Live migration

· FCoE or iSCSI

· Intracluster communication and CSV

In a converged network architecture, LAN and storage traffic utilize Ethernet as the transport. Fibre Channel and iSCSI are possible choices for the converged infrastructure pattern. Although using the customer address over SMB could also be considered a converged architecture, it is broken into a separate design pattern.

The converged pattern refers to either FCoE or iSCSI approaches. Proper network planning is critical in a converged design. Use of Quality of Service (QoS), VLANs, and other isolation or reservation approaches is strongly recommended, so that storage and LAN traffic is appropriately balanced.

14.3 Storage

Storage provides the final component for workload scaling, and as with any workload, it must be designed properly to provide the required performance and capacity for overall fabric scale. In a converged fabric infrastructure, connectivity to the storage uses an Ethernet-based approach, such as iSCSI or FCoE.

Figure 33 provides an overview of the storage infrastructure for the converged pattern.

Figure 42 Storage minimum configuration

14.3.1 Storage Connectivity

For the operating system volume of the parent partition that is using direct-attached storage to the host, an internal SATA or SAS controller is required, unless the design utilizes SAN for all system-storage requirements, including boot from SAN for the host operating system (Fibre Channel and iSCSI boot are supported in Windows Server 2012 R2).

Depending on the storage protocol and devices that are used in the converged storage design, the following adapters are required to allow shared storage access:

· If using Fibre Channel SAN, two or more converged network adapters (CNAs)

· If using iSCSI, two or more 10-gigabit (GB) Ethernet network adapters or iSCSI HBAs

As described earlier, Windows Server 2012 R2 Hyper-V supports the ability to present SAN storage to the guest workloads that are hosted on the fabric infrastructure by using virtual Fibre Channel adapters. Virtual SANs are logical equivalents of virtual network switches within Hyper-V, and each Virtual SAN maps to a single physical Fibre Channel uplink. To support multiple CNAs, a separate Virtual SAN must be created per physical Fibre Channel CNA and mapped exactly to its corresponding physical topology. When configurations use multiple CNAs, MPIO must be enabled within the virtual machine workload itself. Virtual SAN assignment should follow a similar pattern as Hyper-V virtual switch assignment in that, if there are different classifications of service within the SAN, it should be reflected within the fabric.

As discussed in earlier sections, all physical Fibre Channel equipment must support NPIV. Hardware vendors must also provide drivers that display the Designed for Windows logo for all Fibre Channel CNAs, unless drivers are provided in-box. If zoning that is based on physical Fibre Channel switch ports is part of the fabric design, all physical ports must be added to allow for virtual machine mobility scenarios across hosts in the fabric cluster. Although virtual machines can support iSCSI boot, boot from SAN is not supported over the virtual Fibre Channel adapter and should not be considered as part of workload design.

14.3.2 Storage Infrastructure

The key attribute of the storage infrastructure for the converged pattern is the use of a traditional SAN infrastructure, but it is accessed through an Ethernet transport for the fabric, fabric management, and workload layers. As discussed earlier, the primary reason to adopt or maintain this design is to preserve existing investments in a SAN infrastructure or to maintain the current level of flexibility and capabilities that a SAN-based storage-array architecture provides, while consolidating to a single Ethernet network infrastructure.

For Hyper-V failover-cluster and workload operations in a converged infrastructure, the fabric components utilize the following types of storage:

· Operating system: Non-shared physical boot disks (direct-attached storage or SAN) for the fabric management host servers (unless using boot from SAN)

· Cluster witness: Shared witness disk or file share to support the failover cluster quorum

· Cluster Shared Volumes (CSV): One or more shared CSV LUNs for virtual machines (Fibre Channel or iSCSI), as presented by the SAN

· Guest clustering [optional]: Shared Fibre Channel, shared VHDX, or shared iSCSI LUNs for guest clustering

Figure 34 provides a conceptual view of this architecture for the converged pattern.

Figure 43 Converged architecture pattern

As outlined in the overview, fabric and fabric management host controllers require sufficient storage to account for the operating system and paging files. In Windows Server 2012 R2, we recommend that virtual memory be configured as “Automatically manage paging file size for all drives.”

Although boot from SAN from Fibre Channel and iSCSI storage is supported in Windows Server 2012 R2 and Windows Server 2012, it is widely accepted to have storage configured locally per server to provide these capabilities for each server, given the configuration of standard non-converged servers. In these cases, local storage should include two disks that are configured as RAID 1 (mirror) as a minimum, with an optional global hot spare.

To provide quorum for the server infrastructure, we recommend that you utilize a quorum configuration of Node and Disk Majority. To support this, a cluster witness disk is required to support this quorum model. In converged pattern configurations, we recommend that a 1 GB witness disk that is formatted as NTFS be provided for all fabric and fabric management clusters. This provides resiliency and prevents partition-in-time scenarios within the cluster.

As described in earlier sections, Windows Server 2012 R2 provides multiple host access to a shared disk infrastructure through CSV. For converged patterns, the SAN should be configured to provide adequate storage for virtual machine workloads. Because workload virtual disks often exceed multiple gigabytes, we recommend that where it is supported by the workload, dynamically expanding disks be used to provide higher density and more efficient use of storage. Additional SAN capabilities such as thin provisioning of LUNs can assist with the consumption of physical space. However, this functionality should be evaluated to help make sure that workload performance is not affected.

For the purposes of Hyper-V failover clustering, CSV must be configured in Windows as a basic disk that is formatted as NTFS (FAT and FAT32 are not supported for CSV). CSV cannot be used as a witness disk, and they cannot have Windows Data Deduplication enabled. Although supported, ReFS should not be used in conjunction with a CSV with Hyper-V workloads. A CSV has no restrictions in the number of virtual machines that it can support on an individual CSV, because metadata updates on a CSV are orchestrated on the server side and they run in parallel to provide no interruption and increased scalability.

Performance considerations fall primarily on the IOPS that the SAN provides, given that multiple servers from the Hyper-V failover-cluster stream I/O to a commonly shared LUN. Providing more than one CSV to the Hyper-V failover cluster within the fabric can increase performance, depending on the SAN configuration.

To support guest clustering, LUNs can be presented to the guest operating system through iSCSI or Fibre Channel. Configurations for the converged pattern should include sufficient space on the SAN to support a small number of LUNs that support workloads with high availability requirements that must be satisfied within the guest virtual machines and associated applications.

15 Software Defined Infrastructure Architecture Pattern

Key attributes of the Software Defined Infrastructure pattern (previously called the Continuous Availability over SMB Storage pattern) include the use of the SMB 3.02 protocol, and in the case of Variation A, the implementation of the new Scale-Out File Server cluster design pattern in Windows Server 2012 R2.

This section outlines a finished example of a Software Defined Infrastructure pattern that uses Variation A. The following diagram shows the high-level architecture.

Figure 44 Software Defined Infrastructure storage architecture pattern

The design consists of one or more Windows Server 2012 R2 Scale-Out File Server clusters (left) combined with one or more Hyper-V host clusters (right). In this sample design, a shared SAS storage architecture is utilized by the Scale-Out File Server clusters. The Hyper-V hosts store their virtual machines on SMB shares in the file cluster, which is built on top of Storage Spaces and Cluster Shared Volumes.

A key choice in the Software Defined Infrastructure pattern is whether to use InfiniBand or Ethernet as the network transport between the failover clusters that are managed by Hyper-V and the clusters that are managed by the Scale-Out File Server. Currently, InfiniBand provides higher speeds per port than Ethernet (56 Gbps for InfiniBand compared to 10 or 40 GbE). However, InfiniBand requires a separate switching infrastructure, whereas an Ethernet-based approach can utilize a single physical network infrastructure.

15.1 Compute

The compute infrastructure is one of the primary elements that provides fabric scale to support a large number of workloads. In a Software Defined Infrastructure pattern fabric infrastructure, an array of hosts that have the Hyper-V role enabled provide the fabric with the capability to achieve scale in the form of a large-scale failover cluster.

Figure 36 provides an overview of the compute layer of the private cloud fabric infrastructure.

Figure 45 Microsoft Azure poster

With the exception of storage connectivity, the compute infrastructure of this design pattern is similar to the infrastructure of the converged and non-converged patterns. However, the Hyper-V host clusters utilize the SMB protocol over Ethernet or InfiniBand to connect to storage.

15.1.1 Hyper-V Host Infrastructure

The server infrastructure is comprised of a minimum of four hosts and a maximum of 64 hosts in a single Hyper-V failover cluster instance. Although a minimum of two nodes is supported by failover clustering in Windows Server 2012 R2, a configuration at that scale does not provide sufficient reserve capacity to achieve cloud attributes such as elasticity and resource pooling.

Note   The same sizing and availability guidance that is provided in the Hyper-V Host Infrastructure subsection (in the Non-Converged Architecture Pattern section) applies to this pattern.

Figure 46 provides a conceptual view of this architecture for the Software Defined Infrastructure pattern.

Figure 46 Microsoft Azure poster

A key factor in this computer infrastructure is determining whether Ethernet or InfiniBand will be utilized as the transport between the Hyper-V host clusters and the Scale-Out File Server clusters. Another consideration is how RDMA (recommended) will be deployed to support the design.

As outlined in previous sections, RDMA cannot be used in conjunction with NIC Teaming. Therefore, in this design, which utilizes a 10 GbE network fabric, each Hyper-V host server in the compute layer contains four 10 GbE network adapters. One pair is for virtual machine and cluster traffic, and it utilizes NIC Teaming. The other pair is for storage connectivity to the Scale-Out File Server clusters, and it is RDMA-capable.

15.2 Network

When designing the fabric network for the failover cluster for Windows Server 2012 R2 Hyper-V, it is important to provide the necessary hardware and network throughput to provide resiliency and Quality of Service (QoS). Resiliency can be achieved through availability mechanisms, while QoS can be provided through dedicated network interfaces or through a combination of hardware and software QoS capabilities.

Figure 38 provides an overview of the network layer of the private cloud fabric infrastructure.

Figure 47 Network minimum configuration

15.2.1 Host Connectivity

When you are designing the network topology and associated network components of the private cloud infrastructure, certain key considerations apply. You should provide:

· Adequate network port density: Designs should contain top-of-rack switches that have sufficient density to support all host network interfaces.

· Adequate interfaces to support network resiliency: Designs should contain a sufficient number of network interfaces to establish redundancy through NIC Teaming.

· Network Quality of Service: Although the use of dedicated cluster networks is an acceptable way to achieve Quality of Service, utilizing high-speed network connections in combination with hardware- or software-defined network QoS policies provides a more flexible solution.

· RDMA support: For the adapters (InfiniBand or Ethernet) that will be used for storage (SMB) traffic, RDMA support is required.

The network architecture for this design pattern is critical because all storage traffic will traverse a network (Ethernet or InfiniBand) between the Hyper-V host clusters and the Scale-Out File Server clusters.

15.3 Storage

15.3.1 Storage Connectivity

For the operating system volume of the parent partition that is using direct-attached storage to the host, an internal SATA or SAS controller is required—unless the design utilizes SAN for all system storage requirements, including boot from SAN for the host operating system. (Fibre Channel and iSCSI boot are supported in Windows Server 2012 R2 and Windows Server 2012.)

Depending on the storage transport that is utilized for the Software Defined Infrastructure design pattern, the following adapters are required to allow shared storage access:

Hyper-V host clusters:

· 10 GbE adapters that support RDMA

· InfiniBand adapters that support RDMA

Scale-Out File Server clusters:

· 10 GbE adapters that support RDMA

· InfiniBand adapters that support RDMA

· SAS controllers (host bus adapters) for access to shared SAS storage

The number of adapters and ports that are required for storage connectivity between the Hyper-V host clusters and the Scale-Out File Server clusters depends on a variety of size and density planning factors. The larger the clusters and the higher the number of virtual machines that are to be hosted, the more bandwidth and IOPS capacity is required between the clusters.

15.3.2 Scale-Out File Server Cluster Architecture

The key attribute of Variation A and B of the Software Defined Infrastructure design pattern is the usage of Scale-Out File Server clusters in Windows Server 2012 R2 as the “front end” or access point to storage. The Hyper-V host clusters that run virtual machines have no direct storage connectivity. Instead, they have SMB Direct (RDMA)–enabled network adapters, and they store their virtual machines on file shares that are presented by the Scale-Out File Server clusters.

For the PLA patterns, there are two options for the Scale-Out File Server clusters that are required for Variations A and B. The first is the Fast Track “small” SKU, or “Cluster-in-a-Box,” which can serve as the storage cluster. Any validated small SKU can be used as the storage tier for the “medium” IaaS PLA Software Defined Infrastructure Storage pattern. The small SKU is combined with one or more dedicated Hyper-V host clusters for the fabric.

The second option is a larger, dedicated Scale-Out File Server cluster that meets all of the validation requirements that are outlined in the Software Defined Infrastructure Storage section. Figures 48, 49, and 50 illustrate these options.

Figure 48 Software Defined Infrastructure storage options

In the preceding design, a dedicated fabric management cluster and one or more fabric clusters use a Scale-Out File Server cluster as the storage infrastructure.

Figure 49 Another option for fabric management design

In the preceding design, a dedicated fabric management cluster and one or more fabric clusters use a Fast Track “small” (or “Cluster-in-a-Box”) SKU as the storage infrastructure.

15.3.2.1 Cluster-in-a-Box

As part of the Fast Track program, Microsoft has been working with server industry customers to create a new generation of simpler, high-availability solutions that deliver small implementations as a “Cluster-in-a-Box” or as consolidation appliance solutions at a lower price.

In this scenario, the solution is designed as a storage “building block” for the data center, such as a dedicated storage appliance. Examples of this scenario are cloud solution builders and enterprise data centers. For example, suppose that the solution supported Server Message Block (SMB) 3.0 file shares for Hyper-V or SQL Server. In this case, the solution would enable the transfer of data from the drives to the network at bus and wire speeds with CPU utilization that is comparable to Fibre Channel.

In this scenario, the file server is enabled in an office environment in an enterprise equipment room that provides access to a switched network. As a high-performance file server, the solution can support variable workloads, hosted line-of-business (LOB) applications, and data.

The “Cluster-in-a-Box” design pattern requires a minimum of two clustered server nodes and shared storage that can be housed within a single enclosure design or a multiple enclosure design, as shown in Figure 50.

Figure 50 Cluster-in-a-Box

15.3.2.2 Fast Track Medium Scale-Out File Server Cluster

For higher end scenarios in which larger capacity I/O or performance is required, larger multinode Scale-Out File Server clusters can be utilized. Higher performing networks (such as 10 GbE or 56 GbE InfiniBand) can be used between the failover clusters that are managed by Hyper-V and the file cluster.

The Scale-Out File Server cluster design is scaled by adding additional file servers to the cluster. By using CSV 2.0, administrators can create file shares that provide simultaneous access to data files, with direct I/O, through all nodes in a file-server cluster. This provides better utilization of network bandwidth and load balancing of the file server clients (Hyper-V hosts).

Additional nodes also provide additional storage connectivity, which enables further load balancing between a larger number of servers and disks.

In many cases, the scaling of the file server cluster when you use SAS JBOD runs into limits in terms of how many adapters and individual disk trays can be attached to the same cluster. You can avoid these limitations and achieve additional scale by using a switched SAS infrastructure, as described in previous sections.

Figure 51 illustrates this approach. For simplicity, only file-cluster nodes are diagrammed; however, this could easily be four nodes or eight nodes for scale-out.

Figure 51 Medium scale-out file server

Highlights of this design include the SAS switches, which allow a significantly larger number of disk trays and paths between all hosts and the storage. This approach can enable hundreds of disks and many connections per server (for instance, two or more four-port SAS cards per server).

To have resiliency against the failure of one SAS enclosure, you can use two-way mirroring (use minimum of three disks in the mirror for failover clustering and CSV) and enclosure awareness, which requires three physical enclosures. Two-way mirror spaces must use three or more physical disks. Therefore, three enclosures are required with one disk in each enclosure so that the storage pool is resilient to one enclosure failure. For this design, the pool must be configured with the IsEnclosureAware flag, and the enclosures must be certified to use the Storage Spaces feature in Windows Server 2012 R2.

For enclosure awareness, Storage Spaces leverage the array’s failure and identify lights to indicate drive failure or a specific drive’s location within the disk tray. The array or enclosure must support SCSI Enclosure Services (SES) 3.0. Enclosure awareness is independent of a SAS switch or the number of compute nodes.

This design also illustrates a 10 GbE with RDMA design for the file server cluster to provide high bandwidth and low latency for SMB traffic. This could also be InfiniBand if requirements dictate that. Balancing the available I/O storage capacity through the SAS infrastructure to the demands of the failover clusters that are managed by Hyper-V that will be utilizing the file cluster for their storage is key to a good design. An extremely high-performance InfiniBand infrastructure does not make sense if the file servers will have only two SAS connections to storage.

15.3.3 Storage Infrastructure

For Hyper-V failover-cluster and workload operations in a continuous availability infrastructure, the fabric components utilize the following types of storage:

· Operating system: Non-shared physical boot disks (direct-attached storage or SAN) for the file servers and Hyper-V host servers.

· Cluster witness: File share to support the failover cluster quorum for the file server clusters and the Hyper-V host clusters (a shared witness disk is also supported).

· Cluster Shared Volumes (CSV): One or more shared CSV LUNs for virtual machines on Storage Spaces that are backed by SAS JBOD.

· Guest clustering [optional]: Requires iSCSI or shared VHDX. For this pattern, adding the iSCSI Target Server to the file server cluster nodes can enable iSCSI shared storage for guest clustering. However, shared VHDX is the recommended approach because it maintains separation between the consumer and the virtualization infrastructure that is supplied by the provider.

As outlined in the overview, fabric and fabric management host controllers require sufficient storage to account for the operating system and paging files. However, in Windows Server 2012 R2, we recommend that virtual memory be configured as “Automatically manage paging file size for all drives.”

The Sizing of the physical storage architecture for this design pattern is highly dependent on the quantity and type of virtual machine workloads that are to be hosted.

Given the workload, virtual disks often exceed multiple gigabytes. Where it is supported by the workload, we recommend using dynamically expanding disks to provide higher density and more efficient use of storage.

CSV on the Scale-Out File Server clusters must be configured in Windows as a basic disk that is formatted as NTFS. While supported on CSV, ReFS is not recommended and is specifically not recommend for IaaS. In addition, CSV cannot be used as a witness disk, and they cannot have Windows Data Deduplication enabled.

A CSV has no restrictions in the number of virtual machines that it can support on an individual CSV, because metadata updates on a CSV are orchestrated on the server side, and they run in parallel to provide no interruption and increased scalability.

Performance considerations fall primarily on the IOPS that the file cluster provides, given that multiple servers from the Hyper-V failover cluster connect through SMB to a commonly shared CSV on the file cluster. Providing more than one CSV to the Hyper-V failover cluster within the fabric can increase performance, depending on the file cluster configuration.

16 Multi-Tenant Designs

In many private cloud scenarios, and nearly all hosting scenarios, a multi-tenant infrastructure is required. This section illustrates how a multi-tenant fabric infrastructure can be created by using Windows Server 2012 R2 and the technologies described in this Fabric Architecture Guide.

In general, multi-tenancy implies multiple non-related consumers or customers of a set of services. Within a single organization, this could be multiple business units with resources and data that must remain separate for legal or compliance reasons. Most hosting companies require multi-tenancy as a core attribute of their business model. This might include a dedicated physical infrastructure for each hosted customer or logical segmentation of a shared infrastructure by using software-defined technologies.

16.1 Requirements Gathering

The design of a multi-tenant fabric begins with a careful analysis of the business requirements, which will drive the design. In many cases, legal or compliance requirements drive the design approach, which means that a team of several disciplines (for example, business, technical, and legal) should participate in the requirements gathering phase. If specific legal or compliance regimes are required, a plan to ensure compliance and ongoing auditing (internal or non-Microsoft) should be implemented.

To organize the requirements gathering process, an “outside in” approach can be helpful. For hosted services, the end customer or consumer is outside of the hosting organization. Requirements gathering can begin by taking on the persona of the consumer and determining how the consumer will become aware of and be able to request access to hosted services.

Then consider multiple consumers, and ask the following questions:

· Will consumers use accounts that the host creates or accounts that they use internally to access services?

· Is one consumer allowed to be aware of other consumer’s identities, or is a separation required?

Moving further into the “outside in” process, determine whether legal or compliance concerns require dedicated resources for each consumer:

· Can multiple consumers share a physical infrastructure?

· Can traffic from multiple consumers share a common network?

· Can software-defined isolation meet the requirements?

· How far into the infrastructure must authentication, authorization, and accounting be maintained for each consumer (for example, only at the database level, or including the disks and LUNs that are used by the consumer in the infrastructure)?

The following list provides a sample of the types of design and segmentation options that might be considered as part of a multi-tenant infrastructure:

· Physical separation by customer (dedicated hosts, network, and storage)

· Logical separation by customer (shared physical infrastructure with logical segmentation)

· Data separation (such as dedicated databases and LUNs)

· Network separation (VLANs or private VLANs)

· Performance separation by customer (shared infrastructure but guaranteed capacity or QoS)

The remainder of this section describes multi-tenancy options at the fabric level and how those technologies can be combined to enable a multi-tenant fabric.

16.2 Infrastructure Requirements

The aforementioned requirements gathering process should result in a clear direction and set of mandatory attributes that the fabric architecture must contain. The first key decision is whether a shared storage infrastructure or dedicated storage per tenant is required. For a host, driving toward as much shared infrastructure as possible is typically a business imperative, but there can be cases where it is prohibited.

As mentioned in the previous storage sections in this Fabric Architecture Guide, Windows Server 2012 R2 supports a range of traditional storage technologies such as JBOD, iSCSI and Fiber Channel SANs, and converged technologies such as FCoE. In addition, the new capabilities of storage spaces, cluster shared volumes, storage spaces tiering and Scale-Out File Server clusters present a potentially lower cost solution for advanced storage infrastructures.

The shared versus dedicated storage infrastructure requirement drives a significant portion of the design process. If dedicated storage infrastructures per tenant are required, appropriate sizing and minimization of cost are paramount. It can be difficult to scale down traditional SAN approaches to a large number of small- or medium-sized tenants. In this case, the Scale-Out File Server cluster and Storage Spaces approach, which uses shared SAS JBOD, can scale down cost effectively to a pair of file servers and a single SAS tray.

Figure 52 Scale out file server and storage spaces

On the other end of the spectrum, if shared but logically segmented storage is an option, nearly all storage options become potentially relevant. Traditional Fiber Channel and iSCSI SANs have evolved to provide a range of capabilities to support multi-tenant environments through technologies such as zoning, masking, and virtual SANs. With the scalability enhancements in Windows Server 2012 R2 in the storage stack, large-scale shared storage infrastructures that use the Scale-Out File Server cluster and Storage Spaces can also be a cost effective choice.

Although previous sections discussed architecture and scalability, this section highlights technologies for storage security and isolation in multi-tenant environments.

16.3 Multi-Tenant Storage Considerations

16.3.1 SMB 3.0

The Server Message Block (SMB) protocol is a network file sharing protocol that allows applications on a computer to read and write to files and to request services from server programs in a computer network. The SMB protocol can be used on top of TCP/IP or other network protocols.

By using the SMB protocol, an application (or the user of an application) can access files or other resources on a remote server. This allows users to read, create, and update files on the remote server. The application can also communicate with any server program that is set up to receive an SMB client request.

Windows Server 2012 R2 provides the following ways to use the SMB 3.0 protocol:

· File storage for virtualization (Hyper-V over SMB): Hyper-V can store virtual machine files (such as configuration files, virtual hard disk (VHD) files, and snapshots) in file shares over the SMB 3.0 protocol. This can be used for stand-alone file servers and for clustered file servers that use Hyper-V with shared file storage for the cluster.

· SQL Server over SMB: SQL Server can store user database files on SMB file shares. Currently, this is supported with SQL Server 2008 R2 for stand-alone servers.

· Traditional storage for end-user data: The SMB 3.0 protocol provides enhancements to the information worker (client) workloads. These enhancements include reducing the application latencies that are experienced by branch office users when they access data over wide area networks (WANs), and protecting data from eavesdropping attacks.

16.3.1.1 SMB Encryption

A security concern for data that traverses untrusted networks is that it is prone to eavesdropping attacks. Existing solutions for this issue typically use IPsec, WAN accelerators, or other dedicated hardware solutions. However, these solutions are expensive to set up and maintain.

Windows Server 2012 R2 includes encryption that is built-in to the SMB protocol. This allows end-to-end data protection from snooping attacks with no additional deployment costs. You have the flexibility to decide whether the entire server or only specific file shares should be enabled for encryption. SMB Encryption is also relevant to server application workloads if the application data is on a file server and it traverses untrusted networks. With this feature, data security is maintained while it is on the wire.

16.3.1.2 Cluster Shared Volumes

By using Cluster Shared Volumes (CSV), you can unify storage access into a single namespace for ease-of-management. A common namespace folder that contains all CSV in the failover cluster is created at the path C:\ClusterStorage\. All cluster nodes can access a CSV at the same time, regardless of the number of servers, the number of JBOD enclosures, or the number of provisioned virtual disks.

This unified namespace enables high availability workloads to transparently fail over to another server if a server failure occurs. It also enables you to easily take a server offline for maintenance.

Clustered storage spaces can help protect against the following risks:

· Physical disk failures:  When you deploy a clustered storage space, protection against physical disk failures is provided by creating storage spaces with the mirror resiliency type. Additionally, mirror spaces use “dirty region tracking” to track modifications to the disks in the pool. When the system resumes from a power fault or a hard reset event and the spaces are brought back online, dirty region tracking creates consistency among the disks in the pool.

· Data access failures: If you have redundancy at all levels, you can protect against failed components, such as a failed cable from the enclosure to the server, a failed SAS adapter, power faults, or failure of a JBOD enclosure. For example, in an enterprise deployment, you should have redundant SAS adapters, SAS I/O modules, and power supplies. To protect against a complete disk enclosure failure, you can use redundant JBOD enclosures.

· Data corruptions and volume unavailability: The NTFS file system and the Resilient File System (ReFS) help protect against corruption. For NTFS, improvements to the CHKDSK tool in Windows Server 2012 R2 can greatly improve availability. If you deploy highly available file servers, you can use ReFS to enable high levels of scalability and data integrity regardless of hardware or software failures.

· Server node failures: Through the Failover Clustering feature in Windows Server 2012 R2, you can provide high availability for the underlying storage and workloads. This helps protect against server failure and enables you to take a server offline for maintenance without service interruption.

The following are some of the technologies in Windows Server 2012 R2 that can enable multi-tenant architectures.

· File storage for virtualization (Hyper-V over SMB): Hyper-V can store virtual machine files (such as configuration files, virtual hard disk (VHD) files, and snapshots) in file shares over the SMB 3.0 protocol. This can be used for stand-alone file servers and for clustered file servers that use Hyper-V with shared file storage for the cluster.

· SQL Server over SMB: SQL Server can store user database files on SMB file shares. Currently, this is supported by SQL Server 2012 and SQL Server 2008 R2 for stand-alone servers.

· Storage visibility to only a subset of nodes: Windows Server 2012 R2 enables cluster deployments that contain application and data nodes, so storage can be limited to a subset of nodes.

· Integration with Storage Spaces: This technology allows virtualization of cluster storage on groups of inexpensive disks. The Storage Spaces feature in Windows Server 2012 R2 can integrate with CSV to permit scale-out access to data.

16.3.1.3 Security and Storage Access Control

A solution that uses file clusters, storage spaces, and SMB 3.0 in Windows Server 2012 R2 eases the management of large scale storage solutions because nearly all the setup and configuration is Windows based with associated Windows PowerShell support.

If desired, particular storage can be made visible to only a subset of nodes in the file cluster. This can be used in some scenarios to leverage the cost and management advantage of larger shared clusters, and to segment those clusters for performance or access purposes.

Additionally, at various levels of the storage stack (for example, file shares, CSV, and storage spaces), access control lists can be applied. In a multi-tenant scenario, this means that the full storage infrastructure can be shared and managed centrally and that dedicated and controlled access to segments of the storage infrastructure can be designed. A particular customer could have LUNs, storage pools, storage spaces, cluster shared volumes, and file shares dedicated to them, and access control lists can ensure only that tenant has access to them.

Additionally, by using SMB Encryption, all access to the file-based storage can be encrypted to protect against tampering and eavesdropping attacks. The biggest benefit of using SMB Encryption over more general solutions (such as IPsec) is that there are no deployment requirements or costs beyond changing the SMB settings in the server. The encryption algorithm used is AES-CCM, which also provides data integrity validation.

16.4 Multi-Tenant Network Considerations

The network infrastructure is one of the most common and critical layers of the fabric where multi-tenant design is implemented. It is also an area of rapid innovation because the traditional methods of traffic segmentation, such as VLANs and port ACLs, are beginning to show their age, and they are unable to keep up with highly virtualized, large-scale hosting data centers and hybrid cloud scenarios.

The following sections describe the range of technologies that are provided in Windows Server 2012 R2 for building modern, secure, multi-tenant network infrastructures.

16.4.1 Windows Network Virtualization

Hyper-V Network Virtualization provides the concept of a virtual network that is independent of the underlying physical network. With this concept of virtual networks, which are composed of one or more virtual subnets, the exact physical location of an IP subnet is decoupled from the virtual network topology.

As a result, customers can easily move their subnets to the cloud while preserving their existing IP addresses and topology in the cloud so that existing services continue to work unaware of the physical location of the subnets.

Hyper-V Network Virtualization provides policy-based, software-controlled network virtualization that reduces the management overhead that is faced by enterprises when they expand dedicated IaaS clouds, and it provides cloud hosts with better flexibility and scalability for managing virtual machines to achieve higher resource utilization.

An IaaS scenario that has multiple virtual machines from different organizational divisions (referred to as a dedicated cloud) or different customers (referred to as a hosted cloud) requires secure isolation. Virtual local area networks (VLANs), can present significant disadvantages in this scenario.

For more information, see Hyper-V Network Virtualization Overview in the TechNet Library.

VLANs   Currently, VLANs are the mechanism that most organizations use to support address space reuse and tenant isolation. A VLAN uses explicit tagging (VLAN ID) in the Ethernet frame headers, and it relies on Ethernet switches to enforce isolation and restrict traffic to network nodes with the same VLAN ID. As described earlier, there are disadvantages with VLANs, which introduce challenges in large-scale, multi-tenant environments.

IP address assignment   In addition to the disadvantages that are presented by VLANs, virtual machine IP address assignment presents issues, which include:

· Physical locations in the data center network infrastructure determine virtual machine IP addresses. As a result, moving to the cloud typically requires rationalization and possibly sharing IP addresses across workloads and tenants.

· Policies are tied to IP addresses, such as firewall rules, resource discovery, and directory services. Changing IP addresses requires updating all the associated policies.

· Virtual machine deployment and traffic isolation are dependent on the network topology.

When data center network administrators plan the physical layout of the data center, they must make decisions about where subnets will be physically placed and routed. These decisions are based on IP and Ethernet technology that influence the potential IP addresses that are allowed for virtual machines running on a given server or a blade that is connected to a particular rack in the data center. When a virtual machine is provisioned and placed in the data center, it must adhere to these choices and restrictions regarding the IP address. Therefore, the typical result is that the data center administrators assign new IP addresses to the virtual machines.

The issue with this requirement is that in addition to being an address, there is semantic information associated with an IP address. For instance, one subnet might contain given services or be in a distinct physical location. Firewall rules, access control policies, and IPsec security associations are commonly associated with IP addresses. Changing IP addresses forces the virtual machine owners to adjust all their policies that were based on the original IP address. This renumbering overhead is so high that many enterprises choose to deploy only new services to the cloud, leaving legacy applications alone.

Hyper-V Network Virtualization decouples virtual networks for customer virtual machines from the physical network infrastructure. As a result, it enables customer virtual machines to maintain their original IP addresses, while allowing data center administrators to provision customer virtual machines anywhere in the data center without reconfiguring physical IP addresses or VLAN IDs.

Each virtual network adapter in Hyper-V Network Virtualization is associated with two IP addresses:

· Customer address: The IP address that is assigned by the customer, based on their intranet infrastructure. This address enables the customer to exchange network traffic with the virtual machine as if it had not been moved to a public or private cloud. The customer address is visible to the virtual machine and reachable by the customer.

· Provider address: The IP address that is assigned by the host or the data center administrators, based on their physical network infrastructure. The provider address appears in the packets on the network that are exchanged with the server running Hyper-V that is hosting the virtual machine. The provider address is visible on the physical network, but not to the virtual machine.

The customer addresses maintain the customer's network topology, which is virtualized and decoupled from the actual underlying physical network topology and addresses, as implemented by the provider addresses. Figure 53 shows the conceptual relationship between virtual machine customer addresses and network infrastructure provider addresses as a result of network virtualization.

Figure 53 HNV

Key aspects of network virtualization in this scenario include:

· Each customer address in the virtual machine is mapped to a physical host provider address.

· Virtual machines send data packets in the customer address spaces, which are put into an “envelope” with a provider address source and destination pair, based on the mapping.

· The customer address and provider address mappings must allow the hosts to differentiate packets for different customer virtual machines.

As a result, the network is virtualized by the network addresses that are used by the virtual machines.

Hyper-V Network Virtualization supports the following modes to virtualize the IP address:

· Generic Routing Encapsulation   The Network Virtualization using Generic Routing Encapsulation (NVGRE) is part of the tunnel header. This mode is intended for the majority of data centers that deploy Hyper-V Network Virtualization. In NVGRE, the virtual machine’s packet is encapsulated inside another packet. The header of this new packet has the appropriate source and destination PA IP addresses in addition to the virtual subnet ID, which is stored in the Key field of the Generic Routing Encapsulation (GRE) header.

· IP Rewrite    In this mode, the source and the destination CA IP addresses are rewritten with the corresponding PA addresses as the packets leave the end host. Similarly, when virtual subnet packets enter the end host, the PA IP addresses are rewritten with appropriate CA addresses before being delivered to the virtual machines. IP Rewrite is targeted for special scenarios where the virtual machine workloads require or consume very high bandwidth throughput (~10 Gbps) on existing hardware, and the customer cannot wait for NVGRE-aware hardware.

16.4.2 Hyper-V Virtual Switch

The Hyper-V virtual switch is a software-based, layer-2 network switch that is available in Hyper-V Manager when you install the Hyper-V role. The switch includes programmatically managed and extensible capabilities to connect virtual machines to virtual networks and to the physical network. In addition, Hyper-V virtual switch provides policy enforcement for security, isolation, and service levels.

The Hyper-V virtual switch in Windows Server 2012 R2 introduces several features and enhanced capabilities for tenant isolation, traffic shaping, protection against malicious virtual machines, and simplified troubleshooting.

With built-in support for Network Device Interface Specification (NDIS) filter drivers and Windows Filtering Platform (WFP) callout drivers, the Hyper-V virtual switch enables independent software vendors (ISVs) to create extensible plug-ins (known as virtual switch extensions) that can provide enhanced networking and security capabilities. Virtual switch extensions that you add to the Hyper-V virtual switch are listed in the Virtual Switch Manager feature of Hyper-V Manager.

The capabilities provided in the Hyper-V virtual switch mean that organizations have more options to enforce tenant isolation, to shape and control network traffic, and to employ protective measures against malicious virtual machines.

Some of the principal features that are included in the Hyper-V virtual switch are:

· ARP and Neighbor Discovery spoofing protection: Uses Address Resolution Protocol (ARP) to provide protection against a malicious virtual machine that uses spoofing to steal IP addresses from other virtual machines.
Also uses Neighbor Discovery spoofing to provide protection against attacks that can be launched through IPv6. For more information, see Neighbor Discovery.

· DHCP guard protection: Protects against a malicious virtual machine representing itself as a Dynamic Host Configuration Protocol (DHCP) server for man-in-the-middle attacks.

· Port ACLs: Provides traffic filtering, based on Media Access Control (MAC) or Internet Protocol (IP) addresses and ranges, which enables you to set up virtual network isolation.

· Trunk mode to virtual machines: Enables administrators to set up a specific virtual machine as a virtual appliance, and then direct traffic from various VLANs to that virtual machine.

· Network traffic monitoring: Enables administrators to review traffic that is traversing the network switch.

· Isolated (private) VLAN: Enables administrators to segregate traffic on multiple VLANs, to more easily establish isolated tenant communities.

The features in this list can be combined to deliver a complete multi-tenant network design.

16.4.3 Example Network Design

In Hyper-V Network Virtualization, a customer is defined as the owner of a group of virtual machines that are deployed in a data center. A customer can be a corporation or enterprise in a multi-tenant public data center, or a division or business unit within a private data center. Each customer can have one or more customer networks in the data center, and each customer network consists of one or more customer networks with virtual subnets.

Customer network

· Each customer network consists of one or more virtual subnets. A customer network forms an isolation boundary where the virtual machines within a customer network can communicate with each other. As a result, virtual subnets in the same customer network must not use overlapping IP address prefixes.

· Each customer network has a routing domain, which identifies the customer network. The routing domain ID is assigned by data center administrators or data center management software, such as System Center Virtual Machine Manager (VMM). The routing domain ID has a GUID format, for example, “{11111111-2222-3333-4444-000000000000}”.

Virtual subnets

· A virtual subnet implements the Layer 3 IP subnet semantics for virtual machines in the same virtual subnet. The virtual subnet is a broadcast domain (similar to a VLAN). Virtual machines in the same virtual subnet must use the same IP prefix, although a single virtual subnet can accommodate an IPv4 and an IPv6 prefix simultaneously.

· Each virtual subnet belongs to a single customer network (with a routing domain ID), and it is assigned a unique virtual subnet ID (VSID). The VSID is universally unique and may be in the range 4096 to 2^24-2).

A key advantage of the customer network and routing domain is that it allows customers to bring their network topologies to the cloud. The following diagram shows an example where the Blue Corp has two separate networks, the R&D Net and the Sales Net. Because these networks have different routing domain IDs, they cannot interact with each other. That is, Blue R&D Net is isolated from Blue Sales Net, even though both are owned by Blue Corp. Blue R&D Net contains three virtual subnets. Note that the routing domain ID and VSID are unique within a data center.

Figure 54 Microsoft Azure poster

In this example, the virtual machines with VSID 5001 can have their packets routed or forwarded by Hyper-V Network Virtualization to virtual machines with VSID 5002 or VSID 5003. Before delivering the packet to the virtual switch, Hyper-V Network Virtualization will update the VSID of the incoming packet to the VSID of the destination virtual machine. This will only happen if both VSIDs are in the same routing domain ID. If the VSID that is associated with the packet does not match the VSID of the destination virtual machine, the packet will be dropped. Therefore, virtual network adapters with RDID 1 cannot send packets to virtual network adapters with RDID 2.

Each virtual subnet defines a Layer 3 IP subnet and a Layer 2 broadcast domain boundary similar to a VLAN. When a virtual machine broadcasts a packet, this broadcast is limited to the virtual machines that are attached to switch ports with the same VSID. Each VSID can be associated with a multicast address in the provider address. All broadcast traffic for a VSID is sent on this multicast address.

In addition to being a broadcast domain, the VSID provides isolation. A virtual network adapter in Hyper-V Network Virtualization is connected to a Hyper-V switch port that has a VSID ACL. If a packet arrives on this Hyper-V virtual switch port with a different VSID, the packet is dropped. Packets will only be delivered on a Hyper-V virtual switch port if the VSID of the packet matches the VSID of the virtual switch port. This is the reason that packets flowing from VSID 5001 to 5003 must modify the VSID in the packet before delivery to the destination virtual machine.

If the Hyper-V virtual switch port does not have a VSID ACL, the virtual network adapter that is attached to that virtual switch port is not part of a Hyper-V Network Virtualization virtual subnet. Packets that are sent from a virtual network adapter that does not have a VSID ACL will pass unmodified through the Hyper-V Network Virtualization.

When a virtual machine sends a packet, the VSID of the Hyper-V virtual switch port is associated with this packet in the out-of-band (OOB) data. If Generic Routing Encapsulation (GRE) is the IP virtualization mechanism, the GRE Key field of the encapsulated packet contains the VSID.

On the receiving side, Hyper-V Network Virtualization delivers the VSID in the OOB data and the decapsulated packet to the Hyper-V virtual switch. If IP Rewrite is the IP virtualization mechanism, and the packet is destined for a different physical host, the IP addresses are changed from CA addresses to PA addresses, and the VSID in the OOB data is dropped. Hyper-V Network Virtualization verifies a policy and adds the VSID to the OOB data before the packet is passed to the Hyper-V virtual switch.

16.5 Multi-Tenant Compute Considerations

Similar to the storage and network layers, the compute layer of the fabric can be dedicated per tenant or shared across multiple tenants. That decision greatly impacts the design of the compute layer. Two primary decisions are required to begin the design process:

· Will the compute layer be shared between multiple tenants?

· Will the compute infrastructure provide high availability by using failover clustering?

This leads to four high-level design options:

· Dedicated stand-alone server running Hyper-V

· Shared stand-alone server running Hyper-V

· Dedicated Hyper-V failover clusters

· Shared Hyper-V failover clusters

A live migration without shared storage (also known as “shared nothing” live migration) in Windows Server 2012 R2 enables stand-alone servers running Hyper-V to be a viable option when the running virtual machines do not require high availability. A live migration without shared storage also enables virtual machines to be moved from any Hyper-V host running Windows Server 2012 R2 to another host, with nothing required but a network connection (it does not require shared storage).

For hosts that are delivering a stateless application and web hosting services, this may be an option. A live migration without shared storage enables the host to move virtual machines and evacuate hosts for patching without causing downtime to the running virtual machines. However, stand-alone hosts do not provide virtual machine high availability, so if the host fails, the virtual machines are not automatically started on another host.

The decision of using a dedicated vs. a shared Hyper-V host is primarily driven by the compliance or business model requirements discussed previously.

16.5.1 Hyper-V Role

The Hyper-V role enables you to create and manage a virtualized computing environment by using the virtualization technology that is built in to Windows Server 2012 R2. Installing the Hyper-V role installs the required components and optionally installs management tools. The required components include the Windows hypervisor, Hyper-V Virtual Machine Management Service, the virtualization WMI provider, and other virtualization components such as the virtual machine bus (VMBus), virtualization service provider, and virtual infrastructure driver.

The management tools for the Hyper-V role consist of:

· GUI-based management tools: Hyper-V Manager, a Microsoft Management Console (MMC) snap-in, and Virtual Machine Connection (which provides access to the video output of a virtual machine so you can interact with the virtual machine).

· Hyper-V-specific cmdlets for Windows PowerShell. Windows Server 2012 R2 and Windows Server 2012 include the Hyper-V module for Windows PowerShell, which provides command-line access to all the functionality that is available in the GUI, in addition to functionality that is not available through the GUI.

The scalability and availability improvements in Hyper-V allow for significantly larger clusters and greater consolidation ratios, which are key to the cost of ownership for enterprises and hosts. Hyper-V in Windows Server 2012 R2 supports significantly larger configurations of virtual and physical components than previous releases of Hyper-V. This increased capacity enables you to run Hyper-V on large physical computers and to virtualize high-performance, scaled-up workloads.

Hyper-V provides a multitude of options for segmentation and isolation of virtual machines that are running on the same server. This is critical for shared servers running Hyper-V and cluster scenarios where multiple tenants will host their virtual machines on the same servers. By design, Hyper-V ensures isolation of memory, VMBus, and other system and hypervisor constructs between all virtual machines on a host.

16.5.2 Failover Clustering

Failover Clustering in Windows Server 2012 R2 supports increased scalability, continuously available file-based server application storage, easier management, faster failover, automatic rebalancing, and more flexible architectures for failover clusters.

For the purposes of a multi-tenant design, clusters that are managed by Hyper-V can be used in conjunction with the aforementioned clusters that are managed by Scale-Out File Server for an end-to-end Microsoft solution for storage, network, and compute architectures.

16.5.3 Resource Metering

Service providers and enterprises that deploy private clouds need tools to charge business units that they support while providing the business units with the appropriate resources to match their needs. For hosting service providers, it is equally important to issue chargebacks based on the amount of usage by each customer.

To implement advanced billing strategies that measure the assigned capacity of a resource and its actual usage, earlier versions of Hyper-V required users to develop their own chargeback solutions that polled and aggregated performance counters. These solutions could be expensive to develop and sometimes led to loss of historical data.

To assist with more accurate, streamlined chargebacks while protecting historical information, Hyper-V in Windows Server 2012 R2 and Windows Server 2012 provides Resource Metering, which is a feature that allows customers to create cost-effective, usage-based billing solutions. With this feature, service providers can choose the best billing strategy for their business model, and independent software vendors can develop more reliable, end-to-end chargeback solutions by using Hyper-V.

16.5.4 Management

This guide deals only with fabric architecture solutions, and not the more comprehensive topic of fabric management by using System Center 2012 R2. However, there are significant management and automation options for multi-server, multi-tenant environments that are enabled by Windows Server 2012 R2 technologies.

Windows Server 2012 R2 provides management efficiency with broader automation for common management tasks. For example, Server Manager in Windows Server 2012 R2 enables multiple servers on the network to be managed effectively from a single computer.

With the Windows PowerShell 3.0 command-line interface, Windows Server 2012 R2 provides a platform for robust, multi-computer automation for all elements of a data center, including servers, Windows operating systems, storage, and networking. It also provides centralized administration and management capabilities such as deploying roles and features remotely to physical and virtual servers, and deploying roles and features to virtual hard disks, even when they are offline.

[1] Using the following formula, Windows Server 2012 supports a total of 261,120 LUNs per HBA –
(255 LUN ID per target) × (128 targets per bus) × (8 buses per adapter).

[2] Appendix B: Additional Information About Quorum Modes

These are the sessions for the architecture track this year:

DCIM-B210 - How to Develop a Successful Hybrid Cloud Strategy

Monday, May 12 1:15 PM - 2:30 PM

Speaker(s): Eduardo Kassner

We have talked to hundreds of IT professionals in the world's top enterprise IT organizations that are either planning to build or are deploying a private, public, or hybrid cloud. This session shares what was learned, including the common requirements, roles and processes required to have a successful, comprehensive hybrid cloud strategy. We discuss the problems and roadblocks that an IT organization faces on their journey to provide an efficient, automated, elastic and self-service set of business relevant services. Finally, we explain at the conceptual level, the strategy of using the Microsoft offerings of Windows Server 2012 R2, Microsoft System Center 2012 R2, and Windows Azure to build a private and/or hybrid cloud.

DCIM-B211 - Service Provider Datacenter Architecture

Monday, May 12 3:00 PM - 4:15 PM

Speaker(s): Philip Moss

Delivering solutions that meet users' needs for reliability and availability while driving down costs is a challenge faced by service providers and enterprises alike. Windows Server 2012 ushered in a new generation of system architecture and design. Learn how to take advantage of the capabilities in Windows Server 2012 and the enhancements in Windows Server 2012 R2 to design and deliver highly scalable, modular cloud ready datacenter solutions.

DCIM-B212 - The Enterprise: Be Your Own Cloud Provider

Tuesday, May 13 8:30 AM - 9:45 AM

Speaker(s): Philip Moss

Many enterprises face the challenge of delivering services and compute resources to a wide range of internal teams and departments in a timely and elastic way. Users expect fast response times and self-service delivery; in other words, the enterprise needs to think like a service provider. Learn how to use the core capabilities of Windows Server together with the management and automation of Microsoft System Center and Windows Azure Pack to rapidly deploy self-service resources and services to your user base.

DCIM-B214 - How Windows Azure Architectural Patterns Change Approaches to Solutions, Infrastructure, Security, and Software Development

Wednesday, May 14 10:15 AM - 11:30 AM

Speaker(s): Uli Homann

Windows Azure (and its "cousins" in the public cloud) change the way architecture investments and strategies work. This session walks you through the impact of Azure as the new “normal” from an infrastructure design, application and operations architecture perspective. The session also shows how customers can take advantage of the ability in Azure to move workloads from on-premises or service provider environments to take advantage of its unique scalability and global reach—from virtual machines to web workloads. It also outlines new investments in the developer/operations continuum that shape the future of the application platform and resulting solution architectures.

DCIM-B323 - Building a Real-World Hybrid Cloud with Seamless Application Portability

Monday, May 12 4:45 PM - 6:00 PM

Speaker(s): Bruce Gary

You already know the ingredients: Hyper-V, Microsoft System Center, Windows Azure and Microsoft Visual Studio. Now, using real-world customer examples and a comprehensive demo, learn how to weave these technologies together into a practical, end-to-end, turnkey solution. The demo shows you how enterprise IT can create a secure, multi-tenant, policy-driven cloud that spans from your datacenter into Windows Azure. You also learn how application owners and developers can use this cloud to deploy, manage and seamlessly move .NET and Java applications between private and public resources (without code changes!) using a solution from Apprenda and Microsoft.

DCIM-B339 - Architecting Resilient (Cloud) Applications

Tuesday, May 13 3:15 PM - 4:30 PM

Speaker(s): Richard Seroter

The cloud doesn’t magically make applications perform better or avoid failures. Cloud infrastructure can—and will—fail from time to time, and it’s up to the application architect to compose applications in a highly available manner. How can you identify single points of failure, stamp them out, and design applications with exceptional uptime? This session investigates best practices for building highly available applications, discusses "anti-patterns" that limit scalability, and walks through a sample application that leverages the best of what Windows Azure has to offer. See how to assemble applications that use Windows Azure Web Sites, Windows Azure Virtual Machines, Windows Azure Service Bus, and much more.

DCIM-B349 - Software-Defined Storage in Windows Server 2012 R2 and Microsoft System Center 2012 R2

Wednesday, May 14 8:30 AM - 9:45 AM

Speaker(s): Elden Christensen, Hector Linares, Jose Barreto

Regardless of whether you’re building on a private infrastructure, in a hybrid environment, or deploying to a public cloud, there are optimizations you can make in storage and availability that will improve the manageability and performance of your application and environment. Join this session to hear more about the end-to-end scale, performance, and availability improvements coming with Windows Server. We dive into deploying and managing storage using SMB shares, show the improved experience in everyday storage management such as deploying patches to the cloud, and share how to leverage faster live migration when responding to new load demands. Starting with Windows Server 2012, Microsoft offered a complete storage stack from the hardware up, leveraging commodity JBODs surfaced as Virtual Disks via Storage Spaces, hosted by Scale-Out File Server nodes and servicing client requests via SMB 3.0. Now, with major features added in Windows Server 2012 R2 (e.g., Storage Tiering and SMB Redirection), the story gets even better! As a critical piece in the Modern Datacenter (i.e., a Software-Defined Datacenter), SDS plays a crucial role in improving utilization and increasing cost efficiency, scalability, and elasticity. This session empowers you to architect, implement, and monitor this key capability. Come learn how to design, deploy, configure, and automate a storage solution based completely on Windows technologies, as well as how to troubleshoot and manage via the in-box stack and Microsoft System Center. Live demos galore!

DCIM-B352 - Designing Network Infrastructures for Hybrid Cloud

Tuesday, May 13 5:00 PM - 6:15 PM

Speaker(s): Joe Davies

Windows Azure Virtual Networks allow you to build and extend your on-premises IT infrastructure as an organized set of subnets and virtual machines using Windows Azure infrastructure services. Planning and design for using Windows Azure Virtual Networks involves careful analysis of name resolution, IP address management, scalability, routing and incorporation of your computer management infrastructure and then mapping your needs to the current capabilities of Windows Azure. Join us for a detailed discussion of the planning questions, design considerations, and the step-by-step planning process to address your needs and requirements and extend your infrastructure to the cloud.

DEV-B329 - Maximizing Windows Azure for Development and Testing

Tuesday, May 13 1:30 PM - 2:45 PM

Speaker(s): Michael Withrow

Track: Developer Platform and Tools

Session Type: Breakout

Topic: Infrastructure as a Service, Windows PowerShell

It seems like it's getting harder all the time to get the resources you need to build quick and effective DEV/TEST environments. Where are you going to get the hardware you need without breaking the bank? What if I told you that you don't need that hardware anymore and that Azure can be your DEV/TEST Nirvana? It's true! In this session, we present an approach you can use to create a world class DEV/TEST environment which includes the major planning and design decisions you need to make up-front. After we define the design, we'll put it together and demo a number of DEV/TEST scenarios, including an amazing 20 virtual machine SharePoint lab that we'll deploy using only an XML file that calls PowerShell commands and ends up generating an .rdg file. This is definitely cool stuff and the functionality is equally applicable to production scenarios.

DEV-B348 - Practical DevOps for Datacenter Efficiency

Wednesday, May 14 1:30 PM - 2:45 PM

Speaker(s): Richard Seroter

Are your software releases inconsistent, infrequent, and unreliable? Can you deploy software at the rate demanded by your stakeholders? A key aspect of DevOps is aligning people, processes, and technology towards efficient releases through collaboration and automation. Thanks to a new crop of tools that are increasingly embracing Windows and Windows Azure, there has never been a better time to introduce advanced automation to your release process. This session explores the state of DevOps, and walks through example cloud deployments using a wide range of popular technologies such as Puppet and Microsoft System Center.

PCIT-B212 - Design Considerations for BYOD

Tuesday, May 13 10:15 AM - 11:30 AM

Speaker(s): Tom Shinder, Yuri Diogenes

With the proliferation of devices used today, most enterprises are facing one of the biggest dilemmas in technology, which is to enable users to use their own devices, on their own terms, while forcing them to do so in a secure and controlled manner. Enterprises are moving away from the traditional device-centric model, where they control all devices that users are allowed to perform their tasks, to a Bring Your Own Device (BYOD) model. In this presentation we cover the main planning and design considerations for a BYOD world when using Windows Server 2012 R2 as the platform driver to enable secure BYOD computing.

PCIT-B213 - Access Control in BYOD and Directory Integration in a Hybrid Identity Infrastructure

Wednesday, May 14 3:15 PM - 4:30 PM

Speaker(s): Gayana Bagdasaryan

This session is based on the Active Directory Federation Services (AD FS) in Windows Server 2012 R2. It includes an overview of the new practical applications of this version of AD FS, including enabling users to access resources on their personal devices from anywhere and the enhanced access control risk management tools. It also includes an overview and a demo of the use of AD FS in hybrid-based directory synchronization with single sign-on solutions.

The TechEd Content Catalog is live w/more than 600 sessions and labs to choose from! Browse now: https://aka.ms/TE14Content

See you in Houston!

HTH,

Tom Shinder and Yuri Diogenes

Tom Shinder
tomsh@microsoft.com
Knowledge Engineer, DDEC Solutions Group
Follow me on Twitter: https://twitter.com/tshinder
Facebook: https://www.facebook.com/tshinder

Yuri Diogenes
Knowledge Engineer, DDEC Solutions Group

Table of Contents

1.0 Introduction
2.0 Implementation
    2.1 Provision Accounts
    2.2 Configure site-to-site VPN with Windows Azure
    2.3 Deploy Virtual Machines in Windows Azure
    2.4 Deploy an Application in a Hybrid Cloud Infrastructure
    2.5 Configure Windows Azure Load Balancing
    2.6 Deploy System Center Advisor
    2.7 Configure Same Sign-In
    2.8 Configure Windows Azure Active Directory
    2.9 Configure Windows Azure Autoscale
3.0 Summary
4.0 Technologies Discussed

To provide feedback on this article, leave a comment at the bottom of the article or send e-mail to SolutionsFeedback@Microsoft.com. To easily save, edit, or print your own copy of this article, please read How to Save, Edit, and Print TechNet Articles. When the contents of this article are updated, the version is incremented and changes are entered into the change log. The online version is the current version. For the latest information, see the Hybrid Cloud Infrastructure Solution for Enterprise IT article set. This article discusses technologies listed in the Technologies Discussed section of this article.

1.0 Introduction

This article is one of several that are included in an integrated set called Hybrid Cloud Infrastructure Solution for Enterprise IT. Before you read this article, please read Hybrid Cloud Infrastructure Solution for Enterprise IT – Introduction. It provides an overview of the article set, introduces the intended audience, and defines the articles that are contained within the set. This article provides a step-based approach to implement the design that is detailed in Hybrid Cloud Infrastructure Design Considerations in your environment. Although this article implements steps to install and configure the design solution, the steps are written at a level that assumes you already have some familiarity with the technologies that are utilized in the design.

When new technologies are utilized, more detailed implementation steps are also included. To review lower-level implementation steps, you’re encouraged to read the information that is included in the hyperlinks throughout this article. his article is most helpful to those responsible for implementing a hybrid cloud infrastructure or implementing solutions within enterprise IT organizations.  If you’d like to understand all of the relevant individual design configuration options for hybrid cloud infrastructure solutions so that you can determine which options are most appropriate for your own, unique requirements, then it’s recommended that you read the Hybrid Cloud Infrastructure Design Considerations article. While the Hybrid Cloud Infrastructure Design Considerations article details all available Microsoft product and technology design options and considerations for hybrid cloud infrastructure solutions, it does not provide any example designs or recommendations for specific requirements.

Many people will find it helpful to read both the Design article for the Reference Implementation article set targeted to an organization similar to their own, as well as the Design Considerations article for the hybrid cloud infrastructure problem domain. Others will only find it necessary to read one article or the other. Though the two articles are related, there are no dependencies between them.

2.0 Implementation

The high-level implementation steps for the hybrid cloud infrastructure solution that are detailed in Hybrid Cloud Infrastructure Design of this set are summarized in the following table. Each high-level implementation step includes multiple lower-level implementation steps, which are detailed in the remainder of this article.

Back to section 1.0

2.1 Provision Accounts

After you make all your design decisions, it is time to implement your hybrid cloud infrastructure. During this implementation, the first step is to subscribe to Windows Azure. When you access the Windows Azure Pricing page, you will see the options to Try it now or to Buy now. For our Contoso scenario, we will choose the Pay as you Go option, which allows Contoso to have a zero upfront investment, and they can cancel anytime.

Before you subscribe to Windows Azure, it is important to define which account will be the subscription account and which accounts will be authorized to manage the Windows Azure Management Portal. The subscription account includes the credit card information that is used to pay the subscription, which is not necessarily the same account that will be used to administer the environment.

During the provisioning process, you sign up with your Microsoft account (Outlook.com, Hotmail.com, or Windows Live). Contoso created one team account and this account has the credit card information bound to it. After the subscription is completed, Contoso will create co-administrator accounts to enable authorized users to manage the portal. For more information, see section 2.8.2 Create a New User Account. After you sign in with your Microsoft account, you can follow the Windows Azure sign up process to create your account. During this time, you provide the credit card information that will be charged for the subscription.

Back to section 2.0

2.2 Configure Site-to-Site VPN with Windows Azure

After the Windows Azure account is provisioned, you should use the following steps to start the site-to-site virtual private network (VPN) configuration.

Back to section 2.0

2.2.1 Review Prerequisites for Site-to-Site Connectivity with Windows Azure

Ensure that the VPN device that will be used to connect to the Windows Azure Gateway has the following capabilities:

• Full support for IPsec protocol Full support for IKEv2 if the design choice for the site-to-site VPN connectivity is to use dynamic routing (recommended). For more information, see About VPN Devices for Virtual Network.
• A public and unique Internet IP address. For a list of supported VPN devices, see Known compatible VPN devices.

2.2.2 Create a Virtual Network

You must sign in to the Management Portal to perform this task. In the Management Portal, create a custom virtual network that uses the following parameters:

• Affinity group:
  Create a new affinity group for the gateway. For more information, see Importance of Windows Azure Affinity Groups.
• Region:
  Ensure that the datacenter you select is the closest one to your on-premises resources.
• DNS server:
  Type the IP address of the DNS server that will be used for this connectivity. Because the servers that will be running in Windows Azure need to perform name resolution to be part of the domain, this should be the IP address of your on-premises DNS server.
• Site-to-site connectivity:
  Enter the public IP address of the on-premises VPN device that you selected earlier.
• Virtual network address space:
  Use Classless InterDomain Routing (CIDR) notation to specify the private address (for example, 0.0.0.0/8, 172.16.0.0/12, or 192.168.0.0/16) to your network in Windows Azure.

For more information, see Create a Virtual Network in Windows Azure. Contoso decided to use the configuration that is shown in Figure 1.

Figure 1 

When you create a network to allocate your virtual machines and your gateway, consider the following:

• Select a network ID that is different from your on-premises network ID. This is necessary to avoid routing issues between the virtual machines that are located in Windows Azure and the computers that are located on your premises.
• Select a network for your gateway that is within the same address space as the subnet where the virtual machines will reside. In this configuration, Contoso will host all virtual machines on Subnet-1, and the gateway for your site-to-site VPN Gateway will be located in the Gateway network.

For more information about the restrictions and capabilities of a virtual network in Windows Azure, see Virtual Network FAQ.

2.2.3 Configure Windows Azure Gateway

In this step, you will create a gateway to enable the site-to-site connectivity between your on-premises VPN device and Windows Azure. Use the Windows Azure Network Dashboard to create a gateway. The main parameter that must be selected during this configuration is the routing. You can choose between dynamic and static routing. For more information about dynamic and static routing, see About VPN Devices for Virtual Network. After the gateway is created, it can take up to 15 minutes for it to be operational. Then you’ll need to gather the following information, which will be used to configure the on-premises VPN device:

• IP address for the gateway
• Shared key
• VPN device configuration script template (optional)This information can be obtained in the dashboard for your network in the Management Portal.

For more information about how to obtain this information, read Create a Virtual Network for Site-to-Site Cross-Premises Connectivity.

2.2.4 Configure the On-Premises VPN Device

The on-premises VPN device must be configured by using a set of parameters that matches the Windows Azure Gateway. The IPsec connection must have the settings that are described in the following table.

Although Contoso is aware that the recommendation is to use dynamic routing because it leverages the security enhancements in IKEv2, Contoso decided to use static routing. This is due to a limitation with its on-premises VPN device, which only supports IKEv1. It was not in Contoso’s budget to upgrade the on-premises VPN device, so they decided to keep it during this phase of the project. You can get the configuration script for your VPN device from the Management Portal. For more information about routing types and the devices that are compatible with the routing configuration that you select to use in your deployment, see About VPN Devices for Virtual Network.

2.3 Deploy Virtual Machines in Windows Azure

After you create the connectivity between your on-premises VPN device and Windows Azure, you can create virtual machines that will connect to the on-premises resources. The steps in the following table are required to complete this part of the deployment.

Back to section 2.0

2.3.1 Provision the First Virtual Machine

You will use the Windows Azure Portal to create a new virtual machine. Use the option to create the virtual machine from the gallery, and choose the platform image that is appropriate for your scenario. In Contoso’s scenario, the platform image is Windows Server 2012. Some important parameters that will be used during the virtual machine creation are:

• DNS name:
  Choose a unique name for the virtual machine.
• Storage account:
  Select Use Automatic Generated Storage Account.
• Region:
  Select the closest geographic location for the on-premises VPN device.
• Availability set:
  Create a new availability set to group the virtual machines that are deployed across fault domains and update domains. For more information, see Manage the Availability of Virtual Machines.

Contoso is not going to use the remote features in Windows PowerShell, so make sure to disable this option on the last page of the Create a Virtual Machine Wizard. Contoso decided to leave Remote Desktop Protocol (RDP) enabled at the beginning of the process to configure each virtual machine and test the accessibility from the VPN tunnel. This was a safety measure to ensure that the virtual machines were working before they disabled RDP. To disable RDP after you create the virtual machines, see section 2.3.5 Remove Remote Desktop Protocol and Windows PowerShell Access to the Public Network. At this point, the virtual machine will be created. The provisioning process can take up to 10 minutes to complete. For more information about how to create a virtual machine in Windows Azure, see Create a Virtual Machine Running Windows Server.

2.3.2 Promote the Virtual Machine to Be a Domain Controller

After the virtual machine is provisioned, you can use the Windows Azure Management Portal to connect to the virtual machine. When you use this option, a remote desktop connection is initiated to access the virtual machine. Before you promote the server to a domain controller, it is important to perform some validation checks to ensure that connectivity with on-premises resources is working properly. Here are some tests that you should complete at this point:

• Verify that this virtual machine received the IP address of the DNS that you created in  section 2.2.2 Create a Virtual Network.
• Test the basic connectivity with the Domain Controller on-premises by simply typing a ping against the IP address of the domain controller.
• Use the nslookup or ping command by name against resources that are on-premises.

For more information if you have issues during this phase, see Troubleshooting Connectivity between Windows Azure VM and On-Premise resource. After you validate the initial configuration, join this server to the on-premises domain and promote the server to be a domain controller. For more information about how to install Active Directory Domain Services (AD DS) on a server, see Install Active Directory Domain Services (Level 100).Contoso decided to use a global catalog in all domain controllers, so it is necessary to enable the Global Catalog setting in this new domain controller. For more information, see Add or Remove the Global Catalog.

2.3.3 Configure Active Directory Sites and Services

By default, Active Directory Sites and Services has only one site called Default-First-Site-Name. Although this is not a mandatory step, ideally you should create another site to host the servers that are located in Windows Azure. You should also create a new subnet that matches the subnet that you created in Windows Azure Virtual Network. This becomes very important when you have a large environment and you need to optimize replication among domain controllers. After you create the subnet and the site that aligns with the servers located in Windows Azure, you should move the domain controller that is located in Windows Azure from the Default-First-Site-Name to the new site that you created for Windows Azure. Figure 2 shows an example for Contoso’s scenario, with the new site called Cloud. 


Figure 2 

If your company has more than two sites already in place, you should also consider creating site links to them and costs to optimize replication between the domain controllers that are located on-premises with domain controllers that are located in Windows Azure. For more information, see Overview of Active Directory Sites and Services. As stated in Hybrid Cloud Infrastructure Design Considerations Guide for Enterprise IT, the application's web tier is currently hosted in Contoso's perimeter network because these virtual machines accept inbound connections from the Internet. Security configuration for these virtual machines is managed by the Group Policy settings for the perimeter network. They will continue to apply security settings to the virtual machines by using Group Policy. Therefore, they will add the web-tier virtual machines that are hosted in Windows Azure to their perimeter network in AD DS.

2.3.4 Provision the Remaining Virtual Machines

When you create the remaining virtual machines in Windows Azure, ensure that they are connected with the existing virtual machine (which you created previously), and that they have the same region, affinity group, and virtual network settings as the initial virtual machine. Ensure also that they are part of the same cloud service as the other virtual machines. For more information, including information about how to choose the same cloud service, see Create a Virtual Machine Running Windows Server.

Important   For this scenario, avoid using the Quick Create option while creating a new virtual machine because you won’t be able to set up some of the parameters that were mentioned in section 2.3.1 Provision the First Virtual Machine.

Therefore, your new virtual machine might not have connectivity with the other virtual machines that were already instantiated. For Contoso’s scenario, the following virtual machines must be provisioned at this point:

• One additional domain controller to use for redundancy
• Two front-end web servers

2.3.5 Remove Remote Desktop Protocol to the Public Network

After Contoso performed all the necessary tests to validate that access to the virtual machines is successful and that the initial configuration of the operating system is finished, they decided that remote desktop access to all virtual machines that are located in Windows Azure will be available only through site-to-site connectivity. They will accomplish this by using the internal, private IP address of each virtual machine. This enforces secure remote management through the encrypted tunnel that exists between Windows Azure and the on-premises VPN device.

Important   It is possible to disable RDP from public access while you are creating the virtual machines. You can use the Create a Virtual Machine Wizard as explained in section 2.3.1 Provision the First Virtual Machine.

To remove public access to RDP, you need to access the virtual machine through the Windows Azure Management Portal as follows: Click Endpoints, highlight the Remote Desktop row (as shown in Figure 3), and then click Delete in the command bar.

Figure 3

2.4 Deploy an Application in a Hybrid Cloud Infrastructure

The application that will be used by Contoso will be accessed by external clients in addition to internal (on-premises) clients. The front-end web servers will be located in Windows Azure, and the database server is located on premises. The database server is already running SQL Server 2012. The steps in the following table are required to deploy the application in this hybrid environment.

Back to section 2.0

2.4.1 Install Application Server and Web Server Roles on Front-End Servers

Use the Add Roles and Features Wizard in Server Manager to add the Application Server and Web Server (IIS) roles. For more information, see:

• Web Server (IIS) Overview
• Install the Application Server Role on Your Server

Ensure that these roles are installed on both front-end servers. Contoso followed the IIS Security Best Practices to configure both servers. This step should be performed before you configure the application so that the application will be immediately running on a secure platform.

Important   Some of these settings might affect the application’s functionality. We recommend that before you deploy the application for production on a hardening Internet Information Services (IIS) server, you test all the functionalities to see if the security changes will affect the application’s behavior.

After the Application Server and Web Server roles are installed on each front-end server, you need to install the certificate that it will be used by the application on each web server. This certificate could be generated by an internal certification authority or by a trusted third-party public certification authority. Contoso decided to leverage its internal PKI infrastructure to issue a certificate to install on both front-end servers. For more information about how to install a certificate in IIS 8, see Centralized SSL Certificate Support: SSL Scalability and Manageability.

2.4.2 Install and Configure the Application

The application’s configuration will depend on the environment. For Contoso, the requirement is to validate authentication with Active Directory in a secure manner and to access the on-premises database. To validate this environment, Contoso uses a simple ASP.NET MVC4 Web Application. The application was copied to D:\Inetpub\wwwroot (which is another disk separated from the OS in order to improve performance) on both front-end servers. After the files are there, you can use IIS Manager to convert the folder to an application. Basic authentication was enabled in IIS to validate the authentication with the domain. Because the traffic will be encrypted, this method offers no harm to the user’s credential. The application used by Contoso makes a connection with the on-premises SQL Server 2012 to retrieve data and to show the results on the screen. For more information about how to enable basic authentication in IIS 8, see Authentication.

Note   You can use sample applications that are available in the Windows Azure Training Kit (from the Microsoft Download Center) and a sample database from Adventure Works for SQL Server 2012 (from CodePlex).

2.4.3 Configure the Database Server

Contoso is using an existing database server, which is located on premises, to validate this hybrid deployment. The configuration of the database will depend on the application. In Contoso’s case, no additional configuration was necessary because the database was already in production for another application. For more information, see Install SQL Server 2012.

2.5 Configure a Load-Balanced Endpoint

A load-balanced endpoint in Windows Azure is a specific TCP or UDP endpoint that is used by all virtual machines that are contained in a cloud service. The load balancer in Windows Azure maps requests from external clients that are accessing the application from the Internet to Windows Azure Load Balancer, which will map the front-end server that will be used for that request. Use the following steps to configure this feature in Windows Azure.

Back to section 2.0

2.5.1 Add an Endpoint to a Virtual Machine

From the Windows Azure Management Portal, access the first front-end server (virtual machine) that will be used to load balance the web request, and add the endpoint to port 443. Include a name that makes it easy to identify the purpose of this endpoint, and ensure that the protocol in use is TCP and the public port is 443. Although is not mandatory that the private port is the same as the public port, for Contoso’s scenario the private port is the same (443). In Contoso’s scenario, the access to the application will always be through the Internet. This avoids passing traffic through the VPN tunnel, which leaves this tunnel exclusively for application traffic and remote desktop access to the virtual machines. Users will be using the external URL to access the application through a browser, and the load balancer in Windows Azure will handle the traffic. For detailed steps for creating an endpoint, see How to Set Up Communication with a Virtual Machine.

2.5.2 Change DNS

By default, Windows Azure will use the URL, cloudapp.net. However, Contoso decided to create an entry on their external DNS server to support their name schema. For more information, see Configuring a custom domain name for a Windows Azure cloud service.

2.5.3 Validate the Configuration

To validate a configuration for an endpoint, you need to access the public DNS server name of the virtual machine. You can find this name on the virtual machine dashboard in the Management Portal. To validate if the load balancing is correctly happening between servers, you can access the URL from a client workstation to verify if you can access the application. At this point, you could turn off one front-end server and refresh the browser session. If you get an error message that says the service is unavailable, refresh it again and see if you are able to access the application. In Contoso’s scenario, the validation was successful. After one refresh in the web browser, the session was established on the second front-end server.

2.6 Deploy System Center Advisor

Contoso will use Microsoft System Center Advisor as a service to proactively monitor the virtual machines that are located in Windows Azure and to mitigate potential issues due to a server misconfiguration. Use the following steps to deploy System Center Advisor.

Back to section 2.0

2.6.1 Create or Associate a Microsoft Account to System Center Advisor

The first step is to create or associate an existing Microsoft account to the System Center Advisor service. If you already have a Microsoft Outlook.com, Hotmail.com, or Windows Live account, access the System Center Advisor site, and associate this account to a new advisor. If not, create an account and associate it with a new advisor. System Center Advisor will ask you to create a new password if the one you provide doesn’t meet the complexity requirements.

2.6.2 Download the Setup and Registration Certificate

The System Center Advisor Setup Wizard will install gateway and agent files. These files will be later deployed to each target server that you want to monitor. The certificate is used by the setup program to identify your System Center Advisor account. To download the gateway and agent files, sign in to the System Center Advisor portal, and download them from the dashboard. After you download the files, you will copy them to every target server that you want to monitor. In Contoso’s scenario, one of the front-end servers is used as a gateway and the other three servers (the second front-end server and two domain controllers) are configured as agents.

2.6.3 Deploy the Advisor Agent on the Target Servers

To deploy the agent on the target server first you need to install .NET Framework 3.5. If your target server is running Windows Server 2012, you can use the following command to install .NET Framework 3.5 (it requires the Windows Server 2012 DVD or the ISO file): 

dism.exe /online /enable-feature /featurename:NetFX3 /Source:D:\sources\sxs 

When you finish installing this prerequisite, run the AdvisorSetup.exe and follow the wizard to deploy the agent file.

Note   There is also a prerequisite to install System Center 2012 Operations Manager, but you don’t need to do this separately because the System Center Advisor Setup Wizard will do it for you.

During the installation, you will have a chance to choose the following installation options:

• Gateway:
  You should choose one server in your infrastructure to be the gateway. This server will be responsible for gathering information from the agents and sending it to System Center Advisor.
• Agent:
  This option will enable the target server to collect information and send it to the gateway server.

The gateway and agent servers communicate with each other by using port 80. If this port is not already open, it will be automatically enabled by the setup wizard. After the System Center Advisor installation is complete, the System Center Advisor Configuration Wizard will open so that you can finish the customization. During this wizard, you will be able to configure the following settings:

• Gateway Settings (if you are installing a gateway):
  You will inform the location of the Personal Information Exchange (PFX) file that you downloaded in section 2.6.2 Download the Setup and Registration Certificate. You can also configure the proxy settings to restrict connections from this gateway to certain agents.
• Agent Settings (if you are installing an agent):
  You will inform the name of the gateway server (by using the FQDN)

When the System Center Advisor Configuration Wizard for the gateway is complete, you should see a page similar to Figure 4.

Figure 4

2.6.4 Monitor Servers by using System Center Advisor

After you configure the gateway and deploy the agent files on all the virtual machines that you want to monitor, you can manage your environment by using the System Center Advisor portal. Sign in to the System Center Advisor site, and use the options in the left pane to verify alerts, snapshots, change history, servers, and accounts. When you first deploy the agent files in your environment, it is expected to have a delay for some reports. The example shown in Figure 5 is for the Servers option. An error message shows that a new agent was detected, but it is not yet reported to the advisor.

 
Figure 5 

For more information about how to manage your agents from the System Center Advisor portal, see System Center Advisor Help.

2.7 Configure Same Sign-On

To provide a seamless experience for users who are allowed to manage the portal, Contoso decided to implement a same sign-on process by using Active Directory Federation Services (AD FS) with Windows Azure. Use the following steps to enable this process.

Back to section 2.0

2.7.1 Review Same Sign-On Requirements

For this phase of the project, Contoso needs to ensure that the target server that will receive AD FS has a server authentication certificate installed. The subject name of this Secure Socket Layer (SSL) certificate is used to determine the federation service name for each instance of AD FS that you deploy. Contoso decided to choose a subject name that represents the name of its company. Although Contoso has only one AD FS server to deploy now, they understand that to have a more robust Same Sign-On infrastructure, they need to use the Server Farm option, which allows them to later add more servers. Contoso will leverage its internal PKI infrastructure to generate a server authentication certificate, which will be used to secure communications between federation servers, clients, and federation server proxy computers.In the future, when Contoso adds more servers to its infrastructure, it will evaluate the need to use network load balancing (NLB) to provide fault tolerance, high availability, and load balancing across multiple nodes.

2.7.2 Prepare the Infrastructure

Contoso will install AD FS on a dedicated member server running Windows Server 2012. The core steps to prepare the infrastructure for AD FS are:

• Install the Web Server role. For more information, see Web Server (IIS) Overview.
• Obtain and install a server authentication certificate on the default website for AD FS.  For more information, see Centralized SSL Certificate Support: SSL Scalability and Manageability.
• Configure a dedicated service account in AD FS. Although Contoso is starting the implementation with only one server, this step is required if you plan to add more servers. This dedicated service account is necessary to ensure that all resources required by the AD FS farm are granted access to each of the federation servers in the farm.

2.7.3 Install and Configure AD FS

The process to install AD FS in Windows Server 2012 is similar to installing any other role. By installing AD FS, you are enabling the capability of that server to be a federation server, and all the configuration is done after this phase. Contoso installed AD FS on premises. For more information, see Install the AD FS software. After the installation, some important decisions must be made to configure AD FS. Contoso made the following decisions:

• Deployment Type:
  As previously explained, Contoso has plans to expand the number of AD FS servers. For this reason, they selected the Farm Deployment option.
• Certificate selection:
  On the Specify the Federation Service Name page, the AD FS Federation Server Configuration Wizard automatically selects the certificate that it is installed on the default website in IIS. That’s why it is important to perform the steps that are described in section 2.7.2 Prepare the Infrastructure.
• Service Account:
  On the Specify a Service Account page, they select the account that was previously created.
• Download and install Windows PowerShell for single sign-on with AD FS:
  This is a necessary to configure the trust between AD FS and Windows Azure.

For more information about how to install this module, see Manage Windows Azure Active Directory by Using Windows PowerShell. For more information about configuring the AD FS server, see Configure the first federation server in the federation server farm. When you finish the installation, you can validate the AD FS configuration by following the steps described in the Verify that the federation server is operational.

2.7.4 Configure Trust Between AD FS and Windows Azure

Establishing trust between AD FS and Windows Azure is a one-time operation, which means that even if you later add more servers in the AD FS farm, you will not need to perform this step again. For information about how to set up this trust by using Windows PowerShell, see Set up a trust between AD FS and Windows Azure AD.

Important   If for some reason, your AD FS server resides in Windows Azure, ensure that you create an endpoint port (443) for this AD FS server in the Windows Azure Management Portal. This step is necessary to allow the connectivity between Windows Azure and AD FS during the authentication process. If you don’t perform this step, a user who is trying to sign in into the Management Portal might receive the following error message: This page can’t be displayed. For more information, see section 2.5 Configure a Load-Balanced Endpoint.

2.8 Configure Windows Azure Active Directory

Contoso will use Windows Azure Active Directory to provide the same credential experience for users who are allowed to manage the Management Portal. Use the following steps to set up Windows Azure Active Directory:

Back to section 2.0

2.8.1 Add a Custom Domain

The default enterprise directory in a subscription to Windows Azure comes with a basic domain (yourdomain.onmicrosoft.com). To associate the domain that you primarily use in your company to subscription to Windows Azure, you should add a custom domain. During the steps to add a custom domain, the Add Custom Domain Wizard provides the information that you need to create a TXT record in the authoritative DNS server for the domain that you want to customize. In Contoso’s scenario, the administrator created a TXT record under contoso.com zone on the DNS server. The parameters for this record are provided by the wizard. After you create the record, you can click Verify to validate the domain. When the verification is complete, the local Active Directory integration is activated. To check it in Windows Azure, click Default Directory, and then click Directory Integration. This is shown in Figure 6. 


Figure 6 

For more information, see the section Configure the Custom Domain in How to Map CDN Content to a Custom Domain.

2.8.2 Create a New User Account

After the custom domain is created, you should create a new user account in this domain. This user will be the co-administrator of the portal, and the account will be used to perform directory synchronization by using the DirSync tool. To create a new user account, click Active Directory in the left pane of the Management Portal, and then click Default Directory. You will see the dashboard as shown in Figure 7. Under Explore, click Add a user.


Figure 7

When you add a new user to be the co-administrator of your directory, make sure to:

• Add a user in your organization.
• Select the domain (@yourdomain) to be the suffix for the user instead of the default suffix (@yourdomain.onmicrosoft.com).
• Choose the global administrator role.
• Provide an alternative email address for the user, such as @outlook.com.

The Add User Wizard will provide a temporary password when the account is created. Safely send this temporary password to the user. The user must change the password at the first sign-in to the Management Portal. After you create this user account, you can allow the user to be a co-administrator for the Management Portal. For more information, see Add and Remove Co-Administrators for Your Windows Azure Subscriptions.

Important   Assuming that you will use this user account to install the DirSync tool, this password must be changed before the next step.

2.8.3 Download and Install DirSync

The next step is to prepare the directory synchronization, which requires downloading the DirSync tool. The link to download DirSync is provided in the Management Portal. As shown in Figure 6, on the Directory Integration page, click Directory Sync. For more information about the prerequisites for installing this tool, see Prepare for directory synchronization.  In Contoso’s scenario, this tool was downloaded to a member server located on premises. After downloading the tool and reviewing the prerequisites, open the DirSync tool. The Windows Azure Active Directory Sync tool Configuration Wizard will appear as shown in Figure 8.

Figure 8

On the first page of this wizard, you must type the user name that was created in the previous step and the credentials (after the password was changed). As shown in Figure 9, on the Password Synchronization page, select the Enable Password Sync check box to enable directory synchronization.

Figure 9

As shown in Figure 10, when the configuration is finished, the wizard will ask for confirmation to perform the directory synchronization. Click Finish to confirm.

Figure 10

2.8.4 Validate the Configuration

Now that the directory is synchronized, you can create one on-premises Active Directory account, and this account will be synchronized in Windows Azure Active Directory. This will provide the same credential experience for the users. On the Default Directory page in Windows Azure, click Users (see Figure 6). You will see all the accounts—those created in Windows Azure Active Directory and those that were replicated from your on-premises Active Directory account. As shown in Figure 11, the accounts that were replicated will appear under Local Active Directory in the Sourced From column.

Figure 11

For more information if the synchronization is not success, see Verify directory synchronization.

Now you can validate the Same Sign-On scenario between a domain-joined computer and your Microsoft cloud service. Sign-in by using the same name and password that you use for your corporate credentials. Click the Password text box. If Same Sign-On is set up, the password box will be shaded, and you will be redirected as shown in Figure 12.


Figure 12 

2.9 Configure Windows Azure Autoscale

Contoso will take advantage of the autoscale capability in Windows Azure to enhance the overall performance of their front end web servers. To enable autoscale you will select the Cloud Service and then click the name of the cloud service that has the availability set that contains the VMs. The autoscale option is available under scale as shown in Figure 13:

Figure 13 

Contoso has a cloud service called fendweb1 and within this cloud service it has the availability set called FENDSERVERS with 5 small instances.  They will be adjusting the autoscale to CPU based on the settings below:

• Minimum 3 instances of small (1 Core, 1.75 GB Memory)Contoso wants to start with 3 instances since currently those two front end web servers located on-premises are already overloaded
• Target CPU of 65 to 75Contoso wants to work on a safe range, which is based on their baseline collected when this application was running on-premise.

Scale up and down wait time of 30 minutes after last action Contoso wants to make sure that the platform is stable before make any change. It was decided that 30 minutes is a good time to make changes after other actions were done. After finishing the configuration, Contoso’s autoscale options are similar to Figure 14:


Figure 14 

Read How to Scale an Application for more detailed information on the options available for this setting.

Back to section 1.0

3.0 Summary

This article detailed the steps that are required to implement the design that is detailed in Hybrid Cloud Infrastructure Design Considerations of this article set. If you haven’t read that article, and you are interested in understanding the design that the steps in this article explain, you’re encouraged to read it.

4.0 Technologies Discussed

DNS server
Active Directory Federation Services in Windows Server 2012
Active Directory Certificate Services in Windows Server 2012
Web Server (IIS) role in Windows Server 2012
Active Directory Domain Services
Windows Azure Active Directory
Windows Azure Virtual Machines
Windows Azure Cloud Services
Windows Azure Storage
Windows Azure Storage Services
Windows Azure Recovery Service
Windows Azure Virtual Network
SQL Server 2012

5.0 Authors and Reviewers

Author:
Yuri Diogenes - Microsoft

Contributor:
Tom Shinder - Microsoft

Reviewer:
Jim Dial – Microsoft

6.0 Change Log

Published: August 23, 2013
Version: 1.1
Abstract: The Hybrid Cloud Infrastructure Solution for Enterprise IT article set helps you design and implement a hybrid cloud computing infrastructure based on Windows Azure features and capabilities. This article provides a table of contents for the Hybrid Cloud Infrastructure Solution for Enterprise IT reference implementation article set.

Welcome to the Hybrid Cloud Infrastructure Solution for Enterprise IT article set. This article set is a collection of several related articles that, when used together, enable you to design and implement a hybrid cloud infrastructure. This article set details the implementation of a hybrid cloud infrastructure using hardware vendor-agnostic and Microsoft products and technologies. The Hybrid Cloud Infrastructure Solution for Enterprise IT article set is lab-tested solution and we expect that usage of it can significantly decrease the amount of custom design and testing required to implement a hybrid cloud infrastructure solution.

The following articles constitute the Hybrid IT Infrastructure Solution for Enterprise IT reference implementation article set:

• Overview - This article introduces the domain, the audience, the article set, and the articles included in the article set.
• Scenario Definition - This article describes an example, fictitious enterprise IT organization that is implementing a hybrid cloud infrastructure solution.
• Design -This article details which specific products, technologies, and configuration options were selected, out of the hundreds of individual available options, to meet the unique requirements for the example organization defined in the Scenario Definition article.
• Implementation - This article provides a step-wise approach to implementing the design detailed in the Design article in your environment

In addition to the documents in the Hybrid Cloud Solution for Enterprise IT reference implementation article set, you may also be interested in the Hybrid Cloud Design Considerations article. That article provides in-depth information on the design considerations that should to be addressed when designing any hybrid cloud infrastructure. The Design Considerations article contains detailed discussions on the rationale used to create the design outlined in the Design article.

Change Log

Table of Contents

1.0 Introduction
      1.1 Audience
      1.2 Included Articles
2.0 Summary

1.0  Introduction

For many organizations, most of the information technology (IT) capability utilized by their employees is provided by their own IT department.  The organization typically owns physical datacenter space, owns physical servers, owns software licenses, and employs members of an IT department that manage and operate all of these resources.  However, over time datacenters run out of available space, hardware fails or becomes outdated and must be replaced, software versions become outdated and must be upgraded, and IT staff must manage all of the datacenter space, hardware and software.

To start decreasing the need to add more datacenter space, replace hardware, upgrade software, and manage as much of all of it, most organizations have started providing some of the IT capability that employees consume by first consuming the capability from a public cloud service provider, integrating the capability with some of the capability owned and managed by their IT department, and then providing that integrated capability to their employees. This is often referred to as providing hybrid cloud capability.

IT capabilities that an organization may consume from public cloud computing providers are:

• Software services that provide direct end-user capability.  Examples of such services are Microsoft Office 365, Microsoft Dynamics CRM Online, and others from non-Microsoft providers
• Platform services that application developers integrate into applications.  Examples of such services are Windows Azure Service Bus, Windows Azure SQL Database, and others from non-Microsoft providers
• Infrastructure services that allow organizations to run virtual machines on the public provider's infrastructure. Examples of such services are Windows Azure Infrastructure Services, and others from non-Microsoft providers

An organization may integrate the public cloud computing provider's capabilities with its own capabilities by:

• Integrating authentication mechanisms between their own systems and their provider's systems to provide single sign-on capability to the organization's users
• Integrating their network with the provider's network to provide secured communication between the two organizations
• Integrating systems management tools so that the organization can manage the systems at their public provider as they manage their own on-premises systems

This article set provides guidance for integrating infrastructure services provided by Windows Azure Infrastructure Services with an organization's on-premises infrastructure.

Note

For an excellent video review that describes the rationale for, and the goals of the Hybrid Cloud Infrastructure for Enterprise IT project, please see the Solution Overview video.

1.1  Audience

The primary audiences for this article set are enterprise architects and system designers who are interested in understanding the issues that should be considered before integrating public cloud infrastructure services with their on-premises infrastructure systems. System implementers will also be interested in the implementation steps for the sample design included in the article set.

1.2  Included Articles

This article is one of several articles that are included in the Hybrid Cloud Infrastructure Solution for Enterprise IT article set.  Article sets include multiple articles that are expected to be collectively read, like the chapters of a book.  The guidance in this article set can be used to design and implement a hybrid cloud infrastructure solution for enterprise IT. The article set details implementation of a hybrid cloud infrastructure solution using hardware vendor-agnostic and Microsoft products and technologies. Since this article set is lab-tested, usage of it can significantly decrease the amount of custom design and testing required to implement a hybrid cloud infrastructure solution.

The Hybrid Cloud Infrastructure Solution for Enterprise IT article set includes the articles listed below. Each article name is followed by a description of the article, as well as the role within an enterprise IT organization that will likely find the article most useful to them. While the articles are not dependent upon each other, the context for the text in each article listed lower in the list can be found in the article listed above it in the list. Given this, when a role is listed as the primary audience for multiple articles in the list below, the reader is encouraged to read the articles in the order listed. Each article assumes that its reader has a basic understanding of the hybrid cloud infrastructure domain, and has read this article.

Overview (this article):
This article introduces the hybrid cloud problem domain, the audience, the article set, and the articles included in the article set. 

Scenario Definition:
This article describes an example, fictitious enterprise IT organization that is implementing a hybrid cloud infrastructure solution. It details the organization’s current environment, their design objectives and requirements, and their existing environmental policies and constraints that drive the design of any new technology implementation in their environment. Many organizations will find that they have a similar environment with similar requirements and constraints as the example organization. This article is most helpful to those responsible for designing hybrid cloud infrastructure solutions within enterprise IT organizations, since its contents drive the hybrid cloud infrastructure design.

Design:
This article details which specific products, technologies, and configuration options were selected, out of the hundreds of individual available options, to meet the unique requirements for the example organization defined in the Scenario Definition article. This article also details the rationale for why specific design decisions were made. For organizations that have similar requirements and constraints as the example organization, the lab-tested design and rationale in this article can help decrease both the implementation time and risk of implementing a custom hybrid cloud infrastructure solution. This article is most helpful to those responsible for designing hybrid cloud infrastructures or implementing solutions within enterprise IT organizations, since it details an example design, and the rationale for the design.

If, instead of, or in addition to, only understanding one example hybrid cloud infrastructure design, and the rationale for the design options selected for it, you’d like to understand all of the relevant individual design configuration options for hybrid cloud infrastructure solutions so that you can determine which options are most appropriate for your own, unique requirements, then it’s recommended that you read the Hybrid Cloud Infrastructure Design Considerations article. While the Hybrid Cloud Infrastructure Design Considerations article details all available Microsoft product and technology design options and considerations for hybrid cloud infrastructure solutions, it does not provide any example designs or recommendations for specific requirements. Many people will find it helpful to read both the Design article for the RI article set targeted to an organization similar to their own, as well as the Hybrid Cloud Infrastructure Design Considerations articles for the hybrid cloud infrastructure problem domain. Others will only find it necessary to read one article or the other. Though the two articles are related, there are no dependencies between them.

Implementation:
This article provides a step-based approach to implement the design detailed in the Design article in your environment. While this article lists implementation steps to install and configure the solution, the steps are written at a level that assumes you already have some familiarity with the technologies utilized in the design detailed in the Design article. In cases where new technologies are utilized, more detailed implementation steps are also included within the article. To review lower-leveled implementation steps than provided in this article, you’re encouraged to read the information found at the hyperlinks included throughout the article. This article is most helpful to those responsible for implementing hybrid cloud infrastructure solutions within enterprise IT organizations.

2.0  Summary

This article introduced the Hybrid Cloud Infrastructure Solution for Enterprise IT article set and the articles within it. You’re encouraged to read any of the articles within the set that are most applicable to your role within your organization.

Table of Contents

1.0 Introduction
2.0 Organizational Overview
3.0 Problem Definition
4.0 Envisioning
       4.1 Solution Definition
       4.2 Solution Requirements
5.0 Implementation Strategy
6.0 Summary

To provide feedback on this article, leave a comment at the bottom of the article or send e-mail to SolutionsFeedback@Microsoft.com. To easily save, edit, or print your own copy of this article, please read How to Save, Edit, and Print TechNet Articles. When the contents of this article are updated, the version is incremented and changes are entered into the change log. The online version is the current version.

This article includes discussion of solution and public cloud infrastructure service provider requirements, and therefore does not include references to specific technologies.  Separate articles within this article set discuss the technologies that were selected and implemented to meet the requirements defined in this article.

1.0 Introduction

This article is one of several articles that are included in an integrated article set called the Hybrid Cloud Infrastructure Solution for Enterprise IT. If you haven’t already, before reading this article, please read the Overview article within the article set, as it provides an overview of the article set as a whole, introduces the problem domain for the solution, the audience that it is written for, and the articles contained within it.

This article describes a fictitious enterprise IT organization that integrated resources from a public cloud provider with its private cloud infrastructure to enable a hybrid cloud environment. It details the organization’s original IT environment, the business problems they needed to solve, the solution requirements they defined, and the environmental policies and constraints that drove the design of their solution.

Many organizations will find that they have a similar environment with similar requirements and constraints as the organization discussed in this article. This article is most helpful to those responsible for evaluating and selecting public cloud service providers and integrating resources from public cloud service providers with their own private cloud resources.

2.0 Organizational Overview

Contoso is a manufacturing company. It has a main office and several regional offices that are located all over the world.  The regional offices connect to local branch offices. The figure below shows a high level depiction of Contoso's site infrastructure and the connections between the sites.

The IT department had been implementing and maturing its service management processes and had recently started evolving its infrastructure from disparate hardware and software with little automation to a more standardized private cloud infrastructure that would facilitate greater automation for them in the future.  They had used the Cloud Infrastructure Solution for Enterprise IT article set when they started their journey to a private cloud infrastructure.

3.0 Problem Definition

Contoso's IT consumers were becoming increasingly dissatisfied with both the cost of the IT services provided by their IT department, as well as how long it took the IT department to provide new IT capability to them. The management team knew how quickly resources could be provisioned with public cloud service providers, but they also knew that they would not be able to provide new IT capability to consumers on their new private cloud infrastructure as quickly for some time.  

Though they had recently started their private cloud infrastructure implementation, it would be several months before they were able to provision new IT capability as quickly and automatically as they planned due to the time it was taking them to establish and fully implement their processes and automation. To add to this, the Contoso IT management team knew that it was running out of datacenter space and that 25% of its hardware was reaching end-of-life status next fiscal year.  Adding more data center space and replacing servers would increase its cost to provide IT capability even further.

The management team had read Microsoft's The Economics of the Cloud whitepaper. While the management team didn't have enough data to determine whether or not its own cost to operate IT resources was up to ten times more than operating the same IT resources with a public cloud service provider as was presented in the paper, it did suspect that it could operate some of its IT resources for less than the cost of building a new datacenter and replacing its end-of-life hardware. The potential cost savings, coupled with the faster provisioning of IT resources with a public provider, caused them to create a project team to define:

1. Scenarios for which it might use a public cloud infrastructure service provider
2. The effort required to integrate resources from a public cloud provider with its private cloud resources
3. A low-risk application to migrate to a public cloud service provider that would enable the organization to familiarize itself with both the public cloud service provider and the technology necessary to integrate with a public provider 

The management team also expected the project team to configure the necessary infrastructure integration, and complete the migration of the low-risk application to the public cloud service provider. Upon project completion, the management team would evaluate its actual costs, and if it truly proved cost-effective, it would define plans to move more applications to the public cloud infrastructure service provider.

4.0 Envisioning

After the project team was created, it immediately started to define a solution to Contoso's problems.

4.1 Solution Definition

Though the project team knew that it would initially migrate only one application to a public cloud provider and establish basic infrastructure integration between its private cloud and the public cloud provider, it also knew that if all went well, it would likely move additional applications to the public provider in the future.  Given this, the project team defined the following high-level usage scenarios:

• Hosting line of business applications where/when possible
• Enable disaster recovery scenarios where/when possible
• Support lower-cost storage and archival

The project team then defined the following high-level short and longer-term goals:

• Set up organizational-level account and billing with an external provider of cloud infrastructure services, so that its consumers don’t do so at an individual level with personal accounts that they retain when they leave the organization.
• Allow its consumers to provision new virtual machines with the public cloud provider that have capabilities as close to the same capabilities of virtual machines that are provided on its private cloud.
• Allow its consumers to move existing applications that run on the organization’s private cloud into a public cloud infrastructure as a service (IaaS) offering.
• Allow consumers of applications in the organization to resolve names and authenticate to resources that are running on the public cloud provider’s infrastructure, just as they do with resources that are running in their private cloud.
• Enable core security, data access controls, business continuity, disaster recovery, availability and scalability requirements.
• Enable "same sign-on" to both the portal and migrated applications.
• Provide secure network access between on-premises resources and resources at the public cloud infrastructure service provider.

Next, the project team defined the following goals for their pilot project:

• Identify and migrate an existing, low-risk, low business impact line of business application to the public cloud infrastructure service provider
• Configure the foundational infrastructure integration to enable migration of the application
• Migrate the application to the public provider

The project team members quickly identified the company's time card application as their migration candidate.  This was a fairly simple application that employees entered the vacation and sick days that they used into.  It contained low business impact information, and did not maintain any personally identifiable information, so Contoso's IT department thought it made a great pilot migration candidate. The project team currently uses the Cloud Services Foundation Reference Model (CSFRM) as a framework for identifying detailed project requirements in their environment.  

Since Contoso wanted to select a public provider that could meet both their short and longer-term goals, their detailed requirements were what they used to select a public cloud provider.  Due to this, they aligned their requirements to the CSFRM subdomains, and then within each sub-domain, they first divided their requirements into the following categories:

• Public cloud provider requirements: Though Contoso was starting its hybrid infrastructure project with a small solution, it knew the other cloud usage scenarios it expected to utilize in the future.  Therefore, when selecting a provider, Contoso wanted to ensure that the provider met the requirements for both its short-term pilot solution, and the other usage scenarios it planned to utilize the provider for longer-term.  The requirements that you'll find listed in this section include relevant requirements for the provider pilot project solution.
• Provider pilot project solution requirements: These requirements were specific to the time card application that the project team identified as part of its pilot, and were a subset of the broader public cloud provider requirements.  As a result, only the subset of the public cloud provider requirements that are unique to the application are listed in these sections.

Within the two categories of requirements, Contoso IT further divided its detailed solution requirements into groupings that aligned to the CSFRM components within each CSFRM subdomain.  These requirements are listed in the remaining sections of this article.

4.2 Solution Requirements

While the requirements listed below were Contoso’s final requirements, the project team arrived at them only after initially defining all of the requirements listed below, and then adjusting them after incorporating lessons they learned from several different design trial and errors. The requirements were defined to align to Contoso's existing environment, in terms of existing technical capabilities, services, constraints, policies, and processes.

Since Contoso hadn't utilized resources from public cloud providers prior to the start of this project, the project team members first read the Hybrid Cloud Infrastructure Design Considerations article, which helped them better understand the various design considerations that were applicable to designing hybrid cloud infrastructure before they started defining their own requirements. 

All requirements in the following sections are aligned to the applicable subdomains and components of the CSFRM.

4.2.1 Service Delivery Requirements

Before integrating public cloud services with its existing private cloud, Contoso identified the following requirements that were based on its existing IT environment and needs of its consumers. The requirements are aligned to the CSFRM Service Delivery components. As mentioned previously, the requirements below are Contoso's final requirements.  They had started their project by defining an initial list, but found that some requirements negated others, and eventually arrived at the final list below.

In Contoso's pilot phase they had only a single consumer with a single application, which simplified their service delivery requirements due to the limited scope of the pilot phase. Contoso IT will define unique service delivery requirements for each application that it migrates to the public cloud provider in the future.

4.2.1.1 Public Cloud Provider Requirements

Capacity Management

• Able to support scaling to an almost infinite number of virtual machines as demand increases
• Able to support up to 100 Mbps bandwidth to Internet-facing servers
• Able to support up to 100 Mbps bandwidth across the connection between the provider's network and the on-premises network

Availability and Continuity Management

• Able to insure uptime of at least 99.9% for the service components deployed at the public cloud provider
• Provide a service level agreement (SLA) that includes financial penalties if the uptime requirement is not met in any specific month

Information Security Management

• Provide a mechanism to secure the connection between any Internet-based client and any service components contained within the provider's network that accepts those connections
• Provide a mechanism to secure the connection between the provider's network and the on-premises network

Regulatory Policy and Compliance Management

• Support the capabilities and processes required to meet some of Contoso's regulatory and compliance needs.  This is a requirement that Contoso expects to evolve over time as the regulatory and compliance policies that it must comply with evolve. 

Financial Management

• Provide a consumption-based billing capability
• Support the segregation of billing accounts and service management accounts

4.2.1.2 Provider Pilot Project Solution Requirements

Capacity Management

• Support up to four front-end web tier servers
• Support up to 10 Mbps bandwidth to the front-end web tier servers
• Support up to 20 Mbps bandwidth between the front-end web tier servers and the back-end database tier servers

Availability and Continuity Management

• Insure uptime of at least 99.9%

Information Security Management

• Secure the information between front-end web tier servers and the client system browsers using the SSL protocol
• Secure traffic between the front-end web tier servers and the back-end database servers using an IPsec-based security protocol

Financial Management

• Provide a chargeback mechanism that enables Contoso IT visibility into the separate charges they incur for bandwidth, storage and compute
• Enable Contoso IT the ability to create a financial management account that does not have access to the virtual machines contained within the provider's network - the management account must only have access to the provider's financial billing and cost reporting sites and information

4.2.2 Service Operations Requirements

Contoso has a variety of operational processes that are applied to the delivery of all services and technical capabilities within its environment. As a result, it had requirements for how the hybrid cloud infrastructure must apply to and comply with their operational processes. The requirements are aligned to the CSFRM Service Operations components.

4.2.2.1 Public Cloud Provider Requirements

Request Fulfillment

• Provisioning application programming interfaces (APIs) or other integration mechanism that allows Contoso to integrate its private cloud self-service capability with the provider's 

Access Management

• Support identity integration with Contoso's on-premises Active Directory identity repository for management access control so as to enable same sign-on with on-premises identity management systems
• Support identity integration with Contoso's on-premises Active Directory identity repository for access management for the application deployed into the hybrid cloud infrastructure
• Allow different levels of access for the account managers and the service administrators. The account manager should not have automatic access to the public side of the hybrid cloud infrastructure management and configuration interfaces.

Asset and Configuration Management

• Enable assets on the public side of the hybrid cloud infrastructure to be manageable through on-premises Group Policy settings

Systems Administration

• Support administrators of the public cloud components only; this makes it possible to assign users as administrators of the public cloud infrastructure components while not enabling them to manage the on-premises components of the hybrid cloud infrastructure

Change Management

• Provide logs that show what changes have been made to the infrastructure on the provider's end

Knowledge Management

• Provide a knowledge base that Contoso's help desk personnel can use when taking calls from consumers of the hybrid cloud infrastructure

Release and Deployment Management

• Provide pre-configured operating system images for virtual machine deployment
• Provide simple, graphical-based deployment mechanims that can be used to deploy new virtual machines
• Provide command-line methods to manage services that support automation
• Support the ability to upload on-premises operating system images for the creation of new virtual machines

Incident and Problem Management

• Must provide support options that range from no-cost options to different levels of paid support

4.2.2.2 Provider Pilot Project Solution Requirements

Request Fulfillment

• None, as during the pilot phase, Contoso will use its current web-based self-service capability and it will provision resources at the public provider on the consumers' behalf.  In future phases, it plans to integrate its self-service capability with the public cloud provider to enable fully automated deployment of public cloud resources.

Access Management

• Support identity integration with Contoso's on-premises Active Directory identity repository for management access control so that identity management for the hybrid cloud infrastructure can be done through the on-premises identity management systems
• Support identity integration with Contoso's on-premises Active Directory identity repository for access management for the application deployed into the hybrid cloud infrastructure so that identity management for the hybrid cloud infrastructure can be done through the on-premises identity management systems
• Allow different levels of access for the account managers and the service administrators

Asset and Configuration Management

• Enable configuration management through Group Policy
• None, as asset management (virtual machines hosted by the public cloud infrastructure service provider) will be done on a manual basis during the pilot

Systems Administration

• Support administrators of the public cloud components only; this makes it possible to assign users as administrators of the public cloud infrastructure components while not enabling them to manage the on-premises components of the hybrid cloud infrastructure

Change Management

• Log files must be made available that show events related to the creation and deletion of virtual machines and other changes to the public side of the hybrid cloud infrastructure

Knowledge Management 

• None, as Contoso will continue to use its current knowledge management system
• Provide a knowledge base that Contoso's help desk personnel can use when taking calls from consumers of the hybrid cloud infrastructure

Release and Deployment Management

• Provide operating system images for the creation of new virtual machines
• Provide simple, graphical based deployment methods that can be used to deploy new virtual machines

Incident and Problem Management

• Provide a basic free support option, in addition to higher level options, that it may provide for a fee

4.2.3 Management and Support

Every organization uses a variety of technical capabilities to manage and support services in their environment. Contoso's private cloud has a variety of capabilities and constraints. The requirements in the sections that follow are aligned to the CSFRM Management and Support components. 

4.2.3.1 Public Cloud Provider Requirements

Consumer and Provider Portal

• Enable a level of self-service so that acquisition of compute, network and storage does not require human intervention
• Expose any "soft limits" (limit on service acquisition that require an override by the public cloud infrastructure service provider) on service acquisition and provide an online method to exceed any soft limits

 Usage and Billing

• Provide information regarding the monthly costs related to service use
• Provide a way that Contoso IT can assess its current billing, before the end of the month statement

Service Reporting

• Provide service consumption reporting
• Provide service usage reporting for at least the last 30 days
• Provide some level of health reporting that provides information about the state of the virtual machines (on, off, rebooted) over time

Service Monitoring

• Provide a mechanism to inform administrators of a service disruption on the public cloud service provider's service offering
• Enable a level of connectivity to the on-premises network so that current operating system and service monitoring solutions can be extended to the cloud service provider's infrastructure

Configuration Management

• Support the ability to manage virtual machine configuration via on-premises Group Policy settings

Authorization

• Enable a level of role based access control to the extent that a single administrator account can be created that cannot be deleted by other administrators
• Enable the creation of multiple administrator accounts that are authorized to manage the public cloud based components of the service

 Authentication

• Enable on-premises Active Directory accounts to be used for authenticating to the service provider's portal so that Contoso doesn't need to maintain two different authentication repositories
• Provide a way to synchronize the on-premises Active Directory with the service provider's authentication system
• Provide a way to use on-premises accounts to authenticate to the provider's portal without requiring passwords to be synchronized between the systems

 Directory

• Limit non-Active Directory account access to only the service owner account
• Enable on-premises Active Directory accounts to log on to the cloud service provider's portal to manage virtual machines
• Enable a mechanism that allows on-premises Active Directory accounts to log on to the services running on the virtual machines hosted in the public cloud infrastructure service network

Deployment and Provisioning

• Support deployment of new operating systems based on images that the cloud service provider makes available
• Support the ability to upload on-premises virtual disk files that can be used to create new virtual machines on the provider's network
• Provide information about the success or failure of a deployed virtual machine

Network Support

• Provide a built-in name resolution system that allows virtual machines to communicate with each other without support from corporate domain name system (DNS) servers
• Support the ability to place DNS servers on the service provider's network that are integrated with the on-premises DNS servers
• Support a way to connect the on-premises network to the public cloud provider's network
• Support a network connection between the public cloud infrastructure provider's network and the on-premises network of at least 100 Mbps of bandwidth

4.2.3.2 Provider Pilot Project Solution Requirements

Network Support

• Support a network connection between the public cloud infrastructure provider's network and the on-premises network of at least 20 Mbps

4.2.4 Infrastructure Services Capabilities Requirements

Every organization uses a variety of infrastructure technical capabilities, or infrastructure services, or some combination of the two to host IT services. Contoso has a variety of existing infrastructure service capabilities that support services within its environment. As a result, it had requirements for how the hybrid cloud infrastructure must integrate with its existing infrastructure service capabilities. The requirements are aligned to the CSFRM Infrastructure components.

4.2.4.1 Public Cloud Provider Requirements

Compute (Virtual Machine)

• Does not charge for virtual machines that are not turned on
• Provide images for operating systems that will be deployed on the provider's network
• Support uploading of images that were created on-premises that can be used to create new virtual machines on the provider's network
• Support auto-scaling of virtual machines

Network

• Support an encrypted connection between the on-premises network and the public cloud infrastructure service provider's network
• Support some mechanism that will enable connectivity between the on-premises network and the service provider's network that contains the virtual machines
• Provide integrated load balancing capabilities from front-end services hosted on the server provider's network

Storage

• Support virtual machine drives that enable you to turn off and on write caching
• Support maintaining multiple copies of virtual drives in geographically dispersed locations
• Does not delete virtual machine disk files out of storage automatically
• Support virtual disk sizes required by service components migrated to the public provider's infrastructure

5.0 Implementation Strategy

The requirements listed in this article were those that Contoso IT defined during the first phase of their hybrid cloud project. The purpose of the project was to enable Contoso IT to begin to understand the capabilities, limitations, and functionality of a hybrid cloud infrastructure where applications, or portions of applications, would be moved to a public cloud provider's infrastructure. Contoso started very conservatively, as it had a very small team that participated in the project, and because it wanted to get a solid understanding of its public provider before migrating more applications to it. The team began by investigating how to integrate the public provider's core infrastructure functionality with its own before it moved additional applications to the public provider.

To begin to familiarize itself with both the public provider, and its hybrid infrastructure, Contoso deployed its time card application during a pilot phase.  This applications was a low business impact, simple two-tier application, that did not include any personally identifiable information. The application has a front-end web tier and a back-end database tier. The front-end web tier is accessible from the Internet and both managed and unmanaged devices are allowed access to it.  Users authenticate to the application with their corporate Active Directory accounts.

The front-end web tier was originally located in the de-militarized zone (DMZ) of Contoso's private cloud.  The back-end database tier was initially located in the internal network of Contoso's private cloud. In Contoso's pilot project, they migrated the web tier to the public provider, and kept the database tier in its existing location. They configured a virtual network over the Internet between their private cloud datacenter and their provider's datacenter. In addition to supporting web tier to database tier traffic, this connection enabled the application to authenticate users with the corporate Active Directory.

Contoso cataloged what they learned during the pilot phase of the project and plans to use this information to help them define the next phase of their project, where they will integrate more comprehensive management and automation capabilities into their hybrid cloud infrastructure. They also plan to begin identifying applications for migration to the public cloud infrastructure service provider's infrastructure during future phases.

The figure below provides a high level view of Contoso's hybrid cloud infrastructure design, which includes the pilot time card application that they initially migrated to the public provider's infrastructure.

6.0 Summary

This article defined the requirements and constraints of a fictitious enterprise IT organization that integrated resources from a public cloud provider with its private cloud resources to enable a hybrid cloud cloud environment. If you’re interested in reviewing a lab-tested physical design that meets the requirements and constraints listed in this article, you’re encouraged to read the Design article in this article set. The Design article details which specific public provider, products, technologies, and configuration options were selected, out of the hundreds of individual available options, to meet the requirements listed in this article.

The Design article also explains the rationale for why specific design decisions were made. For organizations that have requirements and constraints similar to the organization discussed in this article, the lab-tested design and rationale in the Design article can help decrease both the implementation time and the risk of implementing a hybrid cloud solution.  To understand the full spectrum of design options and considerations available to you when designing hybrid cloud infrastructure, you're encouraged to read the Hybrid Cloud Infrastructure Design Considerations article. You may also choose to refer back to the Overview article in this article set, or to view other cloud architectural and solution guidance at the Cloud and Datacenter Solutions Hub.

7.0 Authors and Reviewers

Authors

• Tom Shinder - Microsoft

Contributors

• Jim Dial - Microsoft
• Yuri Diogenes - Microsoft

Reviewers

• John Dawson - Microsoft

8.0 Change Log

Table of Contents

1.0 Introduction
2.0 Conceptual Design
       2.1 Reference Model
       2.2 Principles
       2.3 Patterns
3.0 Physical Design
       3.1 Overview
       3.2 Service Account Acquisition and Billing
       3.3 Network
       3.4 Storage
       3.5 Compute
       3.6 Management and Support
4.0 Summary 

For the latest information, see the Hybrid Cloud Infrastructure Solution for Enterprise IT article set. To provide feedback on this article, leave a comment at the bottom of this article or send e-mail to SolutionsFeedback@Microsoft.com.  To easily save, edit, or print your own copy of this article, please read How to Save, Edit, and Print TechNet Articles. When the contents of this article are updated, the version is incremented and changes are entered into the change log. The online version is the current version. This article discusses technologies listed in the Technologies Discussed section of this article.

1.0 Introduction

This article is one of several articles that are included in an integrated article set called the Hybrid Cloud Infrastructure Solution for Enterprise IT. If you haven’t already, before reading this article, please read the Overview article within this article set, as it provides an overview of the article set as a whole, introduces the problem domain for the solution, the audience that it is written for, and defines the articles contained within it.

This article details which specific public cloud provider, products, technologies, and configuration options were selected, out of the hundreds of individually available options, as well as why the options were selected to meet the unique requirements for the example organization defined in the Scenario Definition article of the article set.  Consequently, this article assumes that you have already read the Scenario Definition article, so if you have not, please do so before reading this article.

For organizations that have similar requirements and constraints as the fictitious organization defined in the Scenario Definition article, the lab-tested design and rationale in this article can help decrease both the implementation time and risk of implementing a hybrid cloud computing infrastructure solution. This article is most helpful to those responsible for evaluating and selecting public cloud service providers and integrating resources from public cloud service providers with their own private cloud resources.

If instead of, or in addition to, only understanding one example hybrid cloud computing infrastructure design, and the rationale for the design options selected for it, you’d like to understand all of the relevant individual design configuration options for hybrid cloud infrastructure solutions so that you can determine which options are most appropriate for your own, unique requirements, then it’s recommended that you read the Hybrid Cloud Infrastructure Design Considerations article. While the Hybrid Cloud Infrastructure Design Considerations article details all available Microsoft cloud computing services, product and technology design options and considerations for hybrid cloud infrastructure solutions, it does not provide any example designs or recommendations for specific requirements.

Many people will find it helpful to read both the Design article (the article you're reading now), as well as the Design Considerations article for the hybrid cloud infrastructure problem domain. Others will only find it necessary to read one article or the other. Though the two articles are related, there are no dependencies between them.

2.0  Conceptual Design

In the Scenario Definition article of this article set, Contoso IT clearly-defined the requirements for the public cloud provider that they selected, as well as the specific requirements for their pilot project. Next it defined a conceptual architecture that would meet and support its requirements. The conceptual design provided Contoso a vendor-agnostic architectural foundation that helped it ultimately select the best cloud providers and services, products and technologies to implement its conceptual design. The sections that follow detail the different aspects of Contoso’s conceptual design.

2.1 Reference Model

The first design artifact that Contoso decided to create for its hybrid cloud infrastructure was a vendor-agnostic reference model. They wanted the reference model to serve as a common definition of the capabilities and functions that their hybrid cloud infrastructure would provide. Though from Microsoft, Contoso was already using the vendor-agnostic Microsoft Cloud Services Foundation Reference Model (CSFRM) as a foundation for its IT environment as a whole, so it first defined what aspects of the model were applicable to its hybrid cloud infrastructure.

The figure below illustrates the Microsoft CSFRM.  The components with yellow borders are the components that are most applicable to Contoso's hybrid cloud infrastructure. The Physical Design section of this article provides design details for the yellow-bordered components in the figure below.

Note:
Further explanation of the Microsoft CSFRM is not included in this article. If you’re interested in understanding it further however, you’re encouraged to read the Microsoft Cloud Services Foundation Reference Model (CSFRM) article, which is part of the Microsoft Cloud Services Foundation Reference Architecture (CSFRA) article set.

2.2  Principles

After Contoso IT defined its reference model, it defined principles for integrating infrastructure services from a public cloud provider with its private cloud. They defined principles as “guidelines” for their physical designs to adhere to, and knew when they defined them that they were aspirational, because achieving them would take time and effort.  Additionally, they knew that fully achieving some of the principles might require technological offerings that are not yet available.

Just as Contoso had applied the principles from the The Microsoft Cloud Services Foundation Reference Architecture - Principles, Concepts, and Patterns article to their private cloud implementation, they applied the principles for the hybrid cloud problem domain from the Hybrid Cloud Infrastructure Design Considerations article to their hybrid cloud infrastructure, as described in the sections that follow.

2.2.1 Utilize the Public Cloud First

Statement:

Use the public cloud to host service components, unless there is a compelling reason not to.

Rationale:

In most cases, it's easier, faster, and more cost-effective to host service components with public cloud providers than it is to host them on a private cloud. In this article, service components are any physical hardware or software that together, are used to provide a service. Individual service components may support several different services within an organization.  Each service component may be hosted on a private or public cloud. Data from the research in Microsoft's The Economics of the Cloud whitepaper explains that it can be up to 10X more cost-effective to host applications with public cloud providers.

This principle isn't one of the Microsoft Cloud Services Foundation Reference Architecture principles, but Contoso IT defined it for their hybrid infrastructure to help them begin a mindset shift within their organization. For the last several years, their mode of operation has been to ask themselves "Why should we host any service component on the public cloud, and if so, which components?" They wanted to shift the mindset of their staff dramatically to a mode of operation where they ask themselves "Why wouldn't we host every service component on the public cloud, and if not, which components wouldn't we host there and why?" Contoso IT expects that this principle will become easier to achieve over time, as their understanding and confidence in their chosen public service provider grows.

Implications:

Some service components have privacy, regulatory compliance, or both types of requirements that can't be met by public cloud providers. Some applications are designed with application patterns or availability level requirements that aren't supported by some public cloud providers.  Contoso IT does not apply this principle to any application which has any of these constraints.  It will however, re-evaluate such applications over time, since both public cloud infrastructure service provider capabilities and privacy and regulatory compliance requirements change over time.

In the initial phase of their hybrid infrastructure, Contoso IT only applied this principle to new service component implementations, but in later phases, will apply this principle to service components currently hosted on their private cloud too.  When doing so, they'll evaluate additional factors, such as the cost and risk to migrate existing service components to their strategic public cloud service provider.

2.2.2 Perception of Infinite Capacity

Statement:

From the consumer’s perspective, a cloud service should provide capacity on demand, only limited by the amount of capacity the consumer is willing to pay for.

Rationale:

For this and each of the remaining principles in the following sections, Contoso decided it had the same rationale for applying each principle to it's hybrid cloud infrastructure that it did for applying them to its private cloud infrastructure, so it didn't specify any unique rationale for the remaining principles.

Implications:

Contoso IT required that the strategic public service provider it selected support a mechanism to scale its resources, based on its capacity needs.  Further, it required that the provider support Contoso's ability to add this capacity at any time. Contoso IT also required that its public provider support automated and programmatic and scriptable interfaces that allowed its existing on-premises systems to add the capacity. After Contoso's IT initial hybrid infrastructure implementation phase, it will evaluate existing service components and prioritize migration to the public cloud provider for service components that have regular, high-peak demand spikes.

2.2.3 Perception of Continuous Service Availability

Statement:

From the consumer’s perspective, a cloud service should be available on demand from anywhere, on any device, and at any time.

Implications:

Contoso IT required that the strategic public cloud infrastructure service provider it selected meets its availability requirements of at least 99.9%.  To support its business continuity requirements, it required the provider to have datacenters in multiple geographic locations.

2.2.4 Optimization of Resource Usage

Statement:

The cloud should automatically make efficient and effective use of infrastructure resources.

Implications:

In support of its Utilize the Public Cloud First principle, Contoso IT will optimize usage of its private cloud resources for existing service components that are unable to be hosted in a public cloud.  They will host all new service components in the public cloud that are able to be hosted there.  Over time, Contoso IT will prioritize migration of existing service components that can be hosted in the public cloud, based on their other principles.

2.2.5 Incentivize Desired Behavior

Statement:

Enterprise IT service providers must ensure that their consumers understand the cost of the IT resources that they consume so that the organization can optimize its resources and minimize its costs.

Implications:

Contoso IT currently "shows" the consumption costs of its private cloud services to its internal consumers.  Showing the consumption numbers to its business unit consumers has already proved beneficial to Contoso's overall IT costs as it has brought light to unnecessary consumption, and has already helped them to start minimizing such consumption.  As a result, Contoso IT decided that it would also show the costs of its public cloud consumption costs to consumers.

2.2.6 Create a Seamless User Experience

Statement:

Within an organization, consumers should be oblivious as to who the provider of cloud services are, and should have similar experiences with all services provided to them.

Implications:

Contoso IT has spent several years integrating and standardizing its systems to provide seamless user experiences for its users, and didn't want to go back to multiple authentication mechanisms and inconsistent user interfaces when it decided to incorporate resources from a public provider into its hybrid services.  Contoso IT required that the strategic public service provider it selected allow it to integrate authentication and authorization mechanisms with its on-premises services, and also that the provider have network, monitoring, and provisioning mechanisms that allowed Contoso IT to integrate the public provider's systems with its own on-premises systems.

2.3 Patterns

Patterns are specific, reusable ideas that have been proven solutions to commonly occurring problems. Just as Contoso had applied the patterns from the The Microsoft Cloud Services Foundation Reference Architecture - Principles, Concepts, and Patterns article to their private cloud implementation, they applied the patterns for the hybrid cloud problem domain from the Hybrid Cloud Infrastructure Design Considerations article to their hybrid cloud infrastructure, as described in the sections that follow.

2.3.1 Resource Pooling

Problem: When dedicated infrastructure resources are used to support each service independently, their capacity is typically underutilized. This leads to higher costs for both the provider and the consumer.

Solution: Contoso IT has existing resource pools for its private cloud.  Contoso's existing private cloud is partitioned into separate systems management partition resource pools today, but to date, Contoso has had no reason to create separate service class partitions resource pools.  Contoso IT decided that it would treat the resources at the public provider as a security service class partition resource pool, as security-related concerns would, at least initially, be a key factor in whether service components were hosted at the public cloud infrastructure service provider's network or on the private cloud.

After its hybrid cloud infrastructure implementation, Contoso had two separate service class partition resource pools, one within its private cloud, which would host medium and high business impact information, and one in its public cloud, which would, at least initially, host only low business impact information. As Contoso IT migrates more of its service components to its public provider in the future, it will re-evaluate how it defines its resource pools, and anticipates that at a minimum, it may create additional service class partitions for different performance, availability, and continuity requirements.  It fully expects to leverage public cloud resources to help it enable different service class partition resource pools in the future.

2.3.2 Scale Unit

Problem: Purchasing individual servers, storage arrays, network switches, and other cloud infrastructure resources requires procurement, installation, and configuration overhead for each individual resource.

Solution: In their private cloud, Contoso's compute scale units consist of multiple servers that they purchase pre-configured in a rack from their supplier.  At their public provider however, since they pay for every resource they add and have no wait time for new capacity as they do when adding capacity to their private cloud, the scale unit for their public cloud service components is one virtual machine.  As they near capacity thresholds, they simply add and remove individual virtual machines, as necessary.

2.3.3 Capacity Plan

Problem: Eventually every cloud infrastructure runs out of physical capacity. This can cause performance degradation of services, the inability to introduce new services, or both.

Solution: Contoso IT selected a public cloud infrastructure service provider that offered them, essentially, unlimited capacity.  As a result, the capacity plan for their pubic cloud service components was quite simple.  Initially, they had planned to monitor the utilization of the service component, define a capacity threshold, and when the threshold was met, add more capacity. 

The public cloud provider they selected however, provided them the capability to auto-scale the number of virtual machines used to host their service components based on thresholds that they defined, so they configured this for their public cloud service components.  The provider then auto-scaled, both up and down, the number of virtual machines that hosted Contoso's service components, in accordance with the thresholds that they defined.

2.3.4 Health Model

Problem: If any component used to provide a service fails, it can cause performance degradation or unavailability of services.

Solution: Initially, Contoso IT perceived the definition of health models for hybrid services to be more difficult than defining health models for services where all components were hosted on premises.  Part of this was due to their fear of the unknown.  They believed they had more control over everything on premises than they did with their public cloud provider, and they also knew more about how everything worked in their private cloud than they did with their public cloud provider.  After they had defined multiple hybrid service health models though, it occurred to them that they didn't need to understand how all the hardware at their public provider worked, and that was actually a benefit to them.  This made their health model definition easier.

Contoso IT selected a public cloud infrastructure service provider that gave them with a service level agreement (SLA), that included an availability level that their provider committed to meet each month.  Contoso IT soon realized that as long as their provider met this SLA, they weren't extremely concerned with how their provider met the SLA, only that it did meet the SLA. 

This allowed much simpler service health models for its hybrid services than it did services that included components that only ran on Contoso's private cloud infrastructure.  In its first phase, Contoso was using the infrastructure as a service functionality from its public provider.  The provider also offered a platform as a service (PaaS) capability, which Contoso planned to utilize in the future.  Contoso expects that their health model definition will be even simpler when using PaaS, as there will be even less functionality they will need to manage and remediate.

2.3.5 Application

Problem: Not all applications are optimized for cloud infrastructures and may not be able to be hosted on cloud infrastructures.

Solution: Initially, Contoso IT decided that it would only migrate application service components to the public cloud that were designed with the stateless Application pattern. They knew that their public provider's infrastructure was designed with the Upgrade Domain and Physical Fault Domain patterns, so they knew that their stateless applications would continue to be resilient in the event of individual virtual machine failures at their provider.

2.3.6 Cost Model

Problem: Consumers tend to use more resources than they really need if there's no cost to them for doing so.

Solution: Contoso IT will show its internal consumers their consumption costs for public cloud resources, just as it does for its private cloud resources, to help it meet its Incentivize Desired Behavior principle.  They will not show the public provider's direct cost to their consumers however.  Instead, they will add additional costs to the public provider's costs, before showing the cost to their internal consumers.  This is because Contoso IT provides additional functionality to the public cloud virtual machines. 

This additional functionality has a cost.  Contoso IT currently has a per virtual machine cost for single sign-on functionality that is incorporated into the cost of its private cloud virtual machines.  It will add this same cost to its public cloud virtual machines.  This combined cost is what it will show to its internal consumers.  In the future, as Contoso adds backup capability, monitoring capability, and other capability to the public cloud resource it consumes, it will further increase the cost of public cloud virtual machine consumption for its internal consumers accordingly.

3.0 Physical Design

Contoso found the Hybrid Cloud Design Considerations article helpful when defining their physical architecture too.  Though the article did discuss Microsoft technologies, it also helped them get a better understanding of the considerations that they should factor into their hybrid cloud infrastructure design.  The guidance in the article helped them evaluate different cloud providers, cloud services, products, and technologies to determine which would best help them meet their requirements and comply with the conceptual design they created for their hybrid cloud infrastructure.

After evaluating several providers and their services, they concluded that Windows Azure would best meet their projected long and short-term requirements, as defined in the Scenario Definition article. In addition, Contoso already had a strong relationship with Microsoft, which also influenced their decision to begin the hybrid cloud infrastructure project using Windows Azure Infrastructure Services.

The Hybrid Cloud Design Considerations article helped Contoso understand which cloud services, products, and technologies would best help it implement components from the reference model that it defined.  The cloud services, products, and technologies that they selected to implement their reference model components are listed in the table below.

After selecting the public cloud provider, cloud services, products, and technologies that they would implement their hybrid cloud infrastructure with, Contoso continued their physical design.

3.1 Overview

Contoso IT broke their hybrid cloud infrastructure project into multiple phases. Phase 1 of their project focused on the integration of critical infrastructure components between their private cloud and the public cloud provider, and the migration of a simple line of business application to the public cloud.  In Phase 1 of the project, Contoso IT selected a simple two-tier application to migrate to the public cloud and to validate the necessary infrastructure integration with their public infrastructure service cloud provider. 

Both tiers of the application were previously hosted on their private private cloud infrastructure, but in Phase 1, they migrated the web tier to the public cloud infrastructure service provider, and decided that the database tier would remain in their on-premises.  They felt this would give them an opportunity to validate their infrastructure design and facilitate future application migrations to the public cloud.

The application includes a front-end web tier and a back-end database tier. Contoso classified the application as a low business impact (LBI) application because it didn't contain any personally-identifiable information (PII), nor did it have any regulatory compliance requirements. They chose an LBI application because they wanted to start with a low risk asset as they learn more about planning, designing, implementing and managing a hybrid cloud infrastructure. In subsequent phases of the project, they will integrate more robust management and automation capabilities with their public cloud infrastructure service provider, and migrate additional applications to the public cloud provider.

The figure below is a diagram of the physical design for Contoso's hybrid cloud infrastructure solution that includes the LBI application that they migrated in Phase I.

Contoso IT has already completed Phase 1 of their project.  The sections that follow explain which design design decisions that they made and why they made the decisions that they did. Contoso IT made their design decisions by selecting the design options and considerations from the Hybrid Cloud Design Considerations article that best helped them meet the requirements that they had defined in the Scenario Definition article. The decisions are grouped to align with both the CSFRM and the considerations order listed in the Hybrid Cloud Design Considerations article, as Contoso IT found the order in the article to be a useful design sequence.  With that said, they did iterate through most of their design decisions several times before finalizing them because of their learning experiences in developing the solution.

3.2 Service Account Acquisition and Billing

Contoso had several payment plan options from which they could choose when using Azure Infrastructure services. They decided to begin their hybrid cloud infrastructure project with a "pay as you go" plan. After they determine what their cost model will be, Contoso will move to the "pay monthly for six months" plan.  Once they move into full production they will finalize on the "pay monthly for twelve months" plan. They made these decisions based on wanting to begin the project slowly and decided a conservative approach would work best early on. As they gain more experience, they will move to payment plans that increase their financial level of commitment to the public cloud infrastructure service provider.

To simplify billing management, Contoso IT decided to use a single billing account.  The billing account will be assigned to the Director of Contoso IT. They also decided that the single billing account will pay for multiple subscriptions. The reason for this is that using a single subscription for multiple projects can be challenging from an organizational and billing perspective. The Windows Azure management portal provides no method for viewing only the resources used by a single project, and there is no way to automatically break out billing on a per-project basis. Contoso IT was aware that they could somewhat alleviate organizational issues by giving similar names to all services and resources that are associated with a project (for example, HRHostedSvc, HRDatabase, HRStorage). However, they decided that would not help them with billing.

Due to challenges related to granularity of access, organization of resources, and project billing, Contoso decided to create multiple subscriptions and associate each subscription with a different project. Another reason why Contoso created multiple subscriptions was to separate the development and production environments that they plan to deploy in the future. A development subscription will allow administrative access to developers while the production subscription will allow administrative access only to operations personnel.

Finally, Contoso realized that separate subscriptions would provide greater clarity in billing, greater organizational clarity when managing resources, and greater control over who has administrative access to a project. They were however, aware that this approach can be more costly than using a single subscription for all of their projects.

DESIGN DECISIONS:
-Begin  with a "pay as you go" payment plan
-Use a single billing account managed by the Director of Contoso IT
-Establish separate subscriptions that are based on project
-Begin with a single subscription

3.3 Network

Contoso has a well-designed network infrastructure that is managed by its networking team and they have determined that for the most part, they will have to make very few changes to their underlying networking infrastructure and support model when integrating it with a public cloud infrastructure provider's network. There were a few decisions they had to make however, regarding connectivity to resources hosted on the public cloud infrastructure service provider's network. These decisions included:

• Network connectivity between on-premises and off-premises resources
• Inbound connectivity to the public cloud infrastructure service network
• Load balancing of inbound connections to virtual machines in the public cloud infrastructure service provider's network
• Name resolution for the public infrastructure service provider's network

The remainder of this section discusses design decisions Contoso made in each of these areas.

3.3.1 Network Connectivity Between On-Premises and Off-Premises Resources

Contoso needed to connect the corporate network to resources located on Azure Virtual Networks. They have decided to use a site-to-site VPN connection to connect the networks. The reason for this is that Contoso needs full bidirectional connectivity to support connections from the front-end web servers on the Azure Virtual Network to the back-end database servers that are located on-premises.  Additionally, they required enterprise-level name resolution for resources connecting both to and from the Azure Virtual Network and they also required connectivity between domain controllers existing on the Azure Virtual Network and the on-premises domain controllers.

DESIGN DECSIONS:
-Establish a site-to-site VPN between the corporate network and the Azure Virtual Network

3.3.2 Inbound Connectivity to the Public Cloud Infrastructure Service Network

After examining their options for inbound connectivity to the Internet-facing servers on the Azure Virtual Network, they decided that all inbound access for client systems would be done over the Internet. This included client systems that connect from both the Internet and from Contoso's on-premises network. The reason they chose this option was because it simplified their DNS configuration, as they would not need to deploy a split DNS infrastructure.

Contoso dediced that inbound access to resources hosted in the Azure Virtual Network from management systems would not be done over the Internet. Because management access requires access to the operating system themselves, Contoso decided that all management activity must be sourced from management workstations and services within the corporate network. Management workstations and services must be physically located on the corporate network, or they must be connected to the corporate network over VPN or DirectAccess.  This decision was driven by Contoso's security team who required that all management access be done from authorized systems located on the corporate network, as this prevents access from the Internet to the operating systems themselves.

DESIGN DECISIONS:
-All client access to resources located on the Azure Virtual Network was done over the Internet
-All management access to resources located on the Azure Virtual Network was done from the corporate network

3.3.3 Load Balancing of Inbound Connections to Virtual Machines in a Public Infrastructure Service Network

Contoso IT required high availability for services running on the Azure Virtual Network. In order to provide high availability and load balancing for front-end web systems, they decided to use the Azure Virtual Network's built-in load balancing feature. This decision was made based on the fact that all inbound connections from client systems would be done over the Internet and not through the corporate network. Windows Network Load Balancing was considered, but since it's not supported in Azure Virtual Networks and because it requires virtual machines to have more than one IP address (which is not supported in Windows Azure), Windows Network Load Balancing was not used for load balancing.

Another reason Contoso made the decision to use Azure Virtual Network's built-in load balancing feature is because Windows Azure includes an SLA that guarantees 99.95% uptime when two or more instances of the service component are installed (such as two instances of the front-end web servers or two domain controllers). Contoso is aware that Traffic Manager is another option they have available to them for load balancing, but decided not to use it at this time since it was in early preview when they were going through their design process.  They will evaluate it further in subsequent project phases.

DESIGN DECISIONS:
-The Azure Virtual Networks built-in load balancer (load balanced endpoints) was used to enable high availability and load balancing for front-end web servers

3.3.4 Name Resolution for the Public Infrastructure Service Network

Contoso IT needed name resolution support for client systems that access virtual machines on the Azure Virtual Network. Since the decision was made that all client access to cloud based resources would be done over the Internet, they decided that they would use name resolution services that are external to both the Azure Virtual Network and the corporate network. DNS entries for resources on the Azure Virtual Network that are accessible over the Internet are maintained on a public DNS server. Client systems located both on and off premises use the public DNS resource records to access the services located on the Azure Virtual Network, although the client systems on the on-premises network use an internal DNS server that contains the same resource records that the public DNS server contains.

Contoso IT also required name resolution support for management systems so that they could access Azure Virtual Network hosted virtual machines. Since they decided to only allow management access to virtual machines on the Azure Virtual Network through the corporate network, they decided to enter the actual virtual machine names into the corporate DNS database. This decision was also supported by the fact that some or most of the resources hosted on the Azure Virtual Network are domain members, and thus are able to take advantage of Active Directory secure Dynamic DNS.

The figure below shows the DNS name resolution process for internal and external users. In the figure you can see that an external user queries a public DNS server that is responsible for hosting public DNS records for the service. The public DNS server returns the public IP address used to access the service in Windows Azure and then the client application connects to that IP address. The figure also shows an internal user sending a query to an internal DNS server that also hosts the public resource records required to reach the service over the Internet. Note that both the internal and external users receive the same public IP address required to reach the service located on Windows Azure. There is a zone transfer that takes place between the internal DNS server hosting the resource records for the service and the publicly accessible server (zone transfer is not depicted in the figure).

The name resolution process for administrative access to the virtual machines located on the Azure Virtual Network is a bit different. In this case, the figure shows that when an administrator wants to connect to a virtual machine on the Azure Virtual Network (for example, using remote desktop protocol (RDP), the query is sent to their Active Directory integrated DNS server that contains resource records for the actual names of the virtual machines located on the Azure Virtual Network. The IP address returned to the administrator's machine is the actual IP address used by the virtual machine on the Azure Virtual Network. The application then connects to that IP address over the site-to-site VPN connection.

DESIGN DECISIONS:
-Public DNS records were created to allow client Internet access to Azure Virtual Network hosted resources
-Zone transfer was enabled for the public DNS entries between the public DNS server and DNS servers on the corporate network
-Private DNS records were created to allow management access to Azure Virtual Network hosted resources

3.4 Storage

The Contoso intranet has a mature storage infrastructure that includes both Fibre Channel iSCSI SANs and Windows Server 2012 Storage Spaces.  In Phase 1 of their hybrid cloud infrastructure project, Contoso IT had no requirement for using Azure storage outside of using it to store the virtual disk files for their virtual machines.  In future phases however, they fully intend to evaluate implementing some of the hybrid storage scenarios discussed in the Hybrid Cloud Design Considerations article.

3.5 Compute

Contoso needed to make Compute design decisions that center on the virtual machines that were hosted on premises and the Azure Virtual Network. Contoso needed to take into account issues related to the Virtual Machine service provided by Windows Azure.

Contoso made decisions centered around the following issues when designing the hybrid cloud infrastructure’s compute components: 

• Operating system and service images
• On-premises physical and virtual server images and disks
• Virtual disk formats and types
• Virtual machine customization
• Virtual machine access
• Virtual machine service availability
• Virtual machine service scalability

The remainder of this section discusses design decisions Contoso made in each of these areas.

3.5.1 Operating System and Service Images

In Phase 1 of Contoso IT's hybrid cloud infrastructure project, they migrated the front-end web tier of a low business impact (LBI) application to the Azure Virtual Network. Previously both the front-end web and database tiers were hosted on virtual machines on their corporate network. The application was custom-developed by Contoso's in-house developers. Contoso IT needed to decide if they wanted to move these virtual machines to the Azure Virtual Network or create new virtual machines on an Azure Virtual Network and then install the applications on those new virtual machines.

Contoso decided to create new virtual machines on an Azure Virtual Network and then install the front-end web tier component of their LBI application on the Azure Virtual Network. They made this decision for three primary reasons:

1. The on-premises virtual machines were using VHDX files and Azure Infrastructure Services did not support VHDX files.  Contoso did not want to convert the VHDX files to VHD files, which Azure Infrastructure Services does support.
2. The front-end web tier component was easy to install and configure
3. They didn't have to wait for their multi-gigabyte virtual machine files to upload into the Azure storage.

In the future, Contoso will move existing virtual machines to an Azure Virtual Network if it finds the overhead of moving the virtual machines to be less than the overhead of creating and configuring new virtual machines in Windows Azure.

Since Azure Infrastructure Services allows its consumers to place their virtual machines in several different datacenters, Contoso decided to place the virtual machines in an Azure datacenter that is closest to the back-end database tier in its private cloud. They made this decision based on their understanding that network performance is best when Azure Infrastructure Services hosted virtual machines are closest to the end-users or servers that they need to communicate with.

For similar performance reasons, Contoso placed all the virtual machines into the same affinity group, as that put all members of the web tier into the same datacenter.

DESIGN DECISIONS:
-Contoso created new virtual machines for the front-end web tier on the Azure Virtual Network
-All virtual machines hosted in Azure Infrastructure Services were made members of the same affinity group
-All virtual machines were placed in an Azure datacenter closest to the on-premises network

3.5.2 On-Premises Physical and Virtual Server Images and Disks Design Decisions

Contoso has organizational standards for server operating systems that host services on their network. They use sysprepped disk images that contain the settings that have been approved by operations and security.  This helps them save time and reduce the risk of configuration errors when deploying new operating systems to virtual machines. However, during the first phase of their hybrid cloud infrastructure project, they decided to create new virtual machines on the Azure Virtual Network to support their LBI application based on the rationale discussed in the Operating System and Service Images section of this article.

For this reason, they decided not to use the "gold" images that are currently deployed on their private cloud. New virtual machines deployed on the Azure Virtual Network were manually configured to comply with corporate standards. In the future they will consider porting their "gold" images to Azure storage and use them as the base for new virtual machines created in Azure Infrastructure Services.

Because the servers in the front-end web tier accepted inbound connections from the Internet, those servers were hosted on Contoso's on-premises DMZ segment prior to moving their roles to the Azure Virtual Network. Security configuration for the web tier server virtual machines was managed by the Group Policy settings for the DMZ organizational unit (OU) in Contoso's Windows Server Active Directory. Contoso continued to apply security settings to the virtual machines via Group Policy after they were moved on to the Azure Virtual Network by placing the the web tier virtual machines hosted on the Azure Virtual Network into their DMZ OU in Active Directory.

DESIGN DECISIONS:
-In the first phase of the hybrid cloud infrastructure project, Contoso did not use corporate "gold" images in Azure Infrastructure Services
-Contoso placed the web tier virtual machines hosted in Azure Infrastructure Services into their DMZ OU

3.5.3 Virtual Disk Formats and Types Design Decisions

Contoso's current cloud infrastructure uses only VHDX files. They are aware that currently Azure Infrastructure Services supports only the VHD format. They are aware that they can perform disk conversion, but they prefer to wait until Azure Infrastructure Services supports the VHDX format so that they do not need to incur the overhead of converting virtual disk files.

Contoso used Microsoft's recommendation to put the operating system on an operating system disk and application data on data disks. Also per Microsoft's recommendation, Contoso decided not to put data on caching disks.

DESIGN DECISIONS:
-Contoso will create new virtual machines on Azure Infrastructure Services and use the native VHD disk format supported by the service
-They will use the Microsoft recommended disk types

3.5.4 Virtual Machine Customization Design Decisions

Contoso's virtual machines are configured with a number of different virtual hardware designs, based on the particular service being supported by the virtual machine or the preferences of the operator who set up the virtual machine. They realized that Azure Infrastructure Services requires users to pick from a number of pre-defined virtual hardware configurations and that the level of customization available on the on-premises cloud infrastructure is not available in Azure Infrastructure Services. Contoso reviewed the virtual hardware configuration for virtual machines running on-premises and mapped these existing configurations to those available on Azure Infrastructure Services. When Contoso moves on-premises services to Azure Infrastructure Services, they will use the mapping table to determine what the size of compute instance they will use to host the service on Azure Infrastructure Services.

DESIGN DECISIONS:
-Contoso mapped current virtual machine hardware configuration to those available in Azure Infrastructure Services
-New virtual machines created on Azure Infrastructure Services were configured based on the hardware mapping
-The front-end web tier virtual machines were deployed with the "small" virtual machine configuration

3.5.5 Virtual Machine and Application Access Design Decisions

Contoso needed a way to access virtual machines on Azure Virtual Networks for management purposes. They decided to use the Remote Desktop Protocol (RDP) and Remote PowerShell to access Windows operating systems, as this is what they use to manage virtual machines on a per server level on corporate network. For Linux-based virtual machines that they plan to deploy in the future, they will use the Secure Shell Protocol (SSH) for remote management.  They will only allow RDP and remote PowerShell access from the corporate network or from machines that are virtually connected from the corporate network.  These connections reach the Azure Virtual Network over their site-to-site VPN connection with the Azure Infrastructure Services.

Azure Infrastructure Services automatically creates a port forwarding rule that enables remote PowerShell and RDP for Windows operating systems and SSH access for Linux operating systems from the Internet to virtual machines created on an Azure Virtual Network. To reduce the attack surface for Internet-connected servers, Contoso security policy prohibits Internet access to the operating systems over remote PowerShell, RDP and SSH. For this reason Contoso IT decided to disable the default port forwarding rule created by Azure Infrastructure Services on all Internet-connected virtual machines.

Access to the application running on the web-tier servers is encrypted with the secure sockets layer (SSL) protocol, since users send corporate credentials to log on to the application. In order to allow this, Contoso IT created port forwarding rules that allowed TCP port 443 to the web-tier servers running on Azure Infrastructure Services. No port forwarding rules were created for the Active Directory domain controllers because no Internet-initiated connections are made to them.

DESIGN DECISIONS:
-RDP was used to perform per server management access to Windows virtual machines on Azure Infrastructure Services Virtual Networks
-The default remote PowerShell and RDP port forwarding rules were removed 
-A port forwarding rule was created that allowed TCP port 443 to all of the front-end web servers

3.5.6 Virtual Machine and Service Availability Design Decisions

Contoso used a hardware load balancer to increase the service availability for their multi-tier applications, typically for the front-end web servers. They wanted to continue to use load balancing technology when moving applications to Azure Infrastructure Services. For this reason, utilized the built-in load balancing feature included in Azure Infrastructure Services.

In addition to using load balancing to help insure service availability, Contoso wanted to make sure that all components were available when the public cloud infrastructure virtualization hosts were being serviced or updated. To solve this problem, Contoso decided that all virtual machines participating in the same service will belong to the same availability set.

DESIGN DECISIONS:
-Contoso used the Azure Infrastructure Services built-in load balancing feature for inbound connections to the service
-All machines participating in the same service were assigned to the same availability set

3.5.7 Virtual Machine Service Scalability

The application Contoso put into Azure Infrastructure Services has two on-premises Internet-connected web tier servers. Contoso IT noticed that when the servers were on-premises, there were times that application performance was sometimes degraded due to the web tier servers being overwhelmed with connection traffic. They had considered adding two more web tier servers when the web tier was hosted on their private cloud to improve application performance, but they had never done so. 

Since Contoso moved the web tier to Azure Infrastructure Services, they tooks advantage of the Azure auto-scaling feature. They created four virtual machines for their web tier, started two of the four, and defined the range of running instances to always be between two and four.  They then configured the auto-scaling feature so that it would add an instance to the tier when processor utilization was more than 80% for more than ten minutes, and then automatically remove instances when processor utilization dropped below 60% for more than ten minutes.  This not only allowed Contoso to meet capacity needs without manual intervention, it saved them costs, as they were only charged for the running instances.

Contoso put all of the web tier servers in its pilot application into the same Cloud Service. The reason for this decision is that auto-scaling virtual machines can only be done within the same service because it's configured within the service.

DESIGN DECISIONS:
-Contoso IT configured the web tier servers of the pilot application to auto-scale between two and four servers
-Auto-scaling parameters were set to a range of two to four servers with a processor utilization range set to 60%-80%
-All servers in the web tier were placed into the same cloud service

3.6 Management and Support

For the servers it hosted with its public cloud provider, Contoso had to make several important design decisions around how it would management and support them. The major design decisions they made were in the following areas:

• Consumer and provider portal
• Usage and billing
• Service reporting
• Infrastructure service provider authentication and authorization
• Application authentication and authorization
• Backup services and disaster recovery

The following sections discuss the design decisions Contoso made and the rationale behind those decisions.

3.6.1 Consumer and Provider Portal

Though Contoso has a private cloud infrastructure, it does not yet have what it considers to be a "private cloud."  For example, it still does not provide its consumers the level of self-service and automation that no longer requires any human intervention. Because of this, they did not provide a consumer-facing self-service portal to their users that automatically triggers the creation of virtual machines.. They plan to do so when they have completed their on-premises private cloud offering. Until then, they will take resource requests via their IT web site where business units can make requests for resources in the private cloud infrastructure.

Contoso needed access to Azure Infrastructure Services in order to provision and configure virtual machines. While they had several options, they decided to begin their hybrid cloud project by using the Azure Portal to manage their Azure Infrastructure Services environment. Contoso was aware that they had other options, such as PowerShell management and System Center App Controller, but they decided that during the first phase of the project, they wanted to keep things as simple as possible. In the next phase of their project, they will seek to create a seamless management experience by using using App Controller or a similar solution. In addition, prior to the next phase of the hybrid cloud project, Contoso will gain experience with PowerShell management of their Azure Infrastructure Services resources.

DESIGN DECISIONS:
-Contoso did not provide a consumer self-service portal to its customers during the first phase of the hybrid cloud infrastructure project
-They used the Azure Portal for the first phase of the project

3.6.2 Usage and Billing

As mentioned in section 3.2 (Service Account Acquisition and Billing Considerations for Public Cloud Infrastructure Design Decisions), Contoso began with a conservative approach in terms of resource acquisition in Azure Infrastructure Services, but plans to systematically grow their financial investment in Azure Infrastructure Services as they gain more experience with the service. In order to both simplify and clarify usage reporting, Contoso used a single billing account that will be responsible for multiple subscriptions. This gave them visibility into usage costs for different service offerings that use the hybrid cloud infrastructure.

During the initial phase of the project, Contoso IT used showback to report the costs to the business group responsible for the application. As they deploy more applications into the hybrid cloud infrastructure, they will continue to use showback with multiple business groups. Over time, they will engage in discussions with the financial leadership at Contoso to determine if chargeback is appropriate for some or all of the business groups that use the hybrid cloud infrastructure.

DESIGN DECISIONS:
-Contoso used a single billing account
-Multiple subscriptions will be used in the future with one subscription per service offering 
-Showback was used instead of chargeback during the initial phase of the project

3.6.3 Service Reporting

Contoso needed reports on service availability for the public cloud infrastructure service side of their hybrid cloud infrastructure. Contoso wanted to provide an SLA to consumers of the hybrid cloud service and reports needed to provide information about uptime for each component of the service as well as the service as a whole. At the time they had a monitoring and reporting system that they used on-premises and the plan is to integrate that system with the Windows Azure reporting system if they find that it is possible. If so, they will carry that out in the next phase of the project.

During the first phase of the hybrid cloud project, Contoso took advantage of the no-cost System Center Advisor software as a service offering from Microsoft to provide service configuration monitoring and reporting that will supplement the reporting capabilities included with Azure Infrastructure Services. Contoso will consider wider adoption of System Center Advisor after they gain more experience with the product during the first phase. In addition, they will use the service dashboard to determine the health of the Azure Infrastructure Services services in the region where the virtual machines are located.

DESIGN DECISIONS:
-Contoso used the Azure Infrastructure Services built-in service reports
-They used System Center Advisor to monitor service components hosted on Azure Infrastructure Services
-They used the service dashboard to monitor the health of the overall Azure Infrastructure Services services in their location

3.6.4 Public Cloud Infrastructure Service Provider Authentication and Authorization

When working with a public cloud infrastructure service provider’s system, Contoso needed to decide what authentication and authorization/access control options they would implement. Design decisions they needed to make in this area included:

• Authentication
• Authorization
• Account management

NOTE:
This section (3.6.4) addresses only authentication, authorization and account management issues for access to the Azure service. This section does not cover authentication, authorization and account management issues related to access to the application itself and the application running on the web tier servers located in Azure Infrastructure Services. The next section (3.6.5) will discuss authentication, authorization and account management issues related to accessing the application.

The following sections discuss the design decisions Contoso made in these areas and the rationale for those decisions.

3.6.4.1 Authentication

Contoso needed to authenticate to the provider’s system to gain access to its system's resources. They considered their options and decided that the most efficient and secure way to provide account access to the Azure Portal and Azure Infrastructure Services hosted resources was to integrate their on-premises Active Directory accounts with Windows Azure. This enabled enterprise level account management since it integrated their on-premises account provisioning and de-provisioning systems with Azure's authentication systems.

Contoso decided that they would synchronize their on-premises authentication system (Active Directory) with Azure by taking advantage of Windows Azure Active Directory, but that they would not synch their on-premises passwords with Windows Azure Active Directory for an extra measure of security. Because they weren't synchronizing their on-premises passwords, they also federated their Active Directory with Windows Azure Active Directory by using Active Directory Federation Services, which enabled a claims-based authentication system that could be used to log onto the Azure Portal.  This allowed them to use the same username and password to sign-in to Windows Azure services that they used to sign-in to their on-premises services, but it allowed them to not need to ever exchange passwords with Windows Azure.

The rational for using Active Directory Federation Service's claims-based authentication mechanism is that since user passwords aren't being synchronized with Windows Azure Active Directory, there has to be some way for Windows Azure to authenticate the user using the on-premises user account. This is accomplished by using DirSync to synchronize the user accounts. Now that Windows Azure Active Directory "knows" about the user accounts, it needs a way to validate the passwords sent by the user. This is accomplished through AD FS. When Windows Azure Active Directory gets a log on request, it sends the claim to the on-premises AD FS server, which is responsible for validating the claim. This enables Contoso to sync their user accounts and not passwords with Windows Azure Active Directory.

After synchronizing their local accounts with Windows Azure Active Directory, Contoso decided to only allow accounts in their Active Directory to have administrative access to Azure Infrastructure Services hosted resources. This means that when when Active Directory accounts are deleted or disabled on-premises, they automatically no longer have access to Windows Azure services.  They believe this functionality will prove very helpful when employees with access to their Windows Azure services leave the company. 

DESIGN DECISIONS:
-Contoso synchronized their on premises Active Directory with Windows Azure Active Directory using DirSync
-Only Active Directory accounts will be allowed administrative access to Azure Infrastructure Services hosted resources for management

3.6.4.2 Authorization and Access Control

Contoso needed to decide what authorization capabilities they would enforce for access to resources contained on-premises and in the public cloud. Since Contoso didn't have a self-service portal (other than their IT web resource request page), its consumers request resources through that web page and their resources are configured for them manually by a member of the IT department. A full self-service portal will be implemented when Contoso completes its private cloud IaaS implementation, and automation will be built into the resource requisitioning process, which will integrate both on-premises and off-premises components.

Contoso reviewed its options for acquiring resources for the public cloud components of the hybrid cloud infrastructure and decided that during the first phase of the project they would just extend their current on-premises approach to requests for Azure Infrastructure Services resources. They accept requests for infrastructure resources through their service desk and Contoso IT deploys components on their on-premises infrastructure or on the public cloud, based on the unique requirements of each application. Only specific members of the Contoso IT group were given administrative access to Azure Infrastructure Services at the first phase of project. Later, these members of the IT group will train the rest of the group on how to manage Azure Infrastructure Services resources.

DESIGN DECISIONS:
-Contoso extended its resource request approach to public cloud components of the hybrid cloud infrastructure
-A subset of the Contoso IT organization were trained on managing Azure Infrastructure Services resources
-The subset of the Contoso IT organization will train the rest of the IT group in the next phase of the project

3.6.4.3 Account Management

Contoso needed an account management system to insure that only valid accounts of authorized users had access to the hybrid cloud infrastructure. They had a mature account management system that was integrated with their on-premises human resources systems. They needed to extend the efficiencies they have with their on-premises system to access to Azure Infrastructure Services hosted resources. As discussed in section 3.6.4.1 (Authentication), they accomplished this goal by integrating their on-premises Active Directory system with Azure Infrastructure Services so that local accounts were required for management access.

Role based access control was important to Contoso security and they implemented this at many levels of their on-premises network. They also required the ability to assign role-based access control to users of their public cloud resources. They decided to assign the head of Contoso IT to the Service Administrator role, and a subset of Contoso IT members to the Co-Administrator role. They did this based on the fact that the Director of Contoso IT makes decisions on who has administrative access and therefore should have the same responsibility for assigning this access to the Azure Infrastructure Services hosted resources.

Contoso was aware that Co-Administrators are allowed to add other Co-Administrators. Therefore, they put a process in place where the Service Administrator is responsible for reviewing the Co-Administrator list on a daily basis to confirm that only authorized users are listed as Co-Administrators. If the Service Administrator is not available due to vacation, sick leave, or any other reason, the Service Administrator's manager will be tasked with this check. The reason for this is that it is possible that a Co-Administrator might add an account that a user could use if they leave the company, which would allow them to retain access privileges when they should no longer have them.

DESIGN DECISIONS:
-On-premises account management was extended to off-premises access to the Azure Portal via claims-based federation
-The Director of Contoso IT was assigned the Service Administrator role
-The Director of Contoso IT will assign Co-Administrator access to a subset of the IT organization
-Contoso IT put a process in place to check on the list of Co-Administrators

3.6.5 Application Authentication and Authorization

Contoso needed to make design decisions around application authentication and authorization. While there are a number of authentication and authorization options available for the applications that Contoso will run in Azure Infrastructure Services, in the majority of cases for Contoso those applications will be dependent on Active Directory. For this reason, Contoso needed to make design decisions for applications that run some or all of their components in Azure Infrastructure Services.

Design decisions for application authentication authorization revolved around the following issues:

• Active Directory domain controller on an Azure Virtual Network
• Domain controllers on an Azure Virtual Network
• Domain Controller Locator 
• Domain, forest and global catalog
• Active Directory name resolution and Geo-location
• Active Directory Federation Services

NOTE:
This section (3.6.5) addresses authentication, authorization and account management issues related to access to the application that has it's web tier hosted in Azure Infrastructure Services. This section does not address issues related to authentication, authorization and account management for access to the Azure services themselves. These issues are discussed in section 3.6.4.

The following sections discuss the design decisions Contoso made in these areas and the rationale for those decisions.

3.6.5.1 Active Directory Domain Controllers in the Public Cloud Infrastructure Provider's Network

Contoso IT needed to decide if they were going to put any domain controllers in an Azure Virtual Network to support applications that require Active Directory for authentication. In addition, they needed to decide what type of domain controller to place on the Azure Virtual Network. Contoso decided to put two domain controllers on the Azure Virtual Network to support user authentication for access to hybrid applications. This allowed users to authenticate even if the site-to-site VPN connection to the on-premises domain controllers failed.

New domain controllers were created on the Azure Virtual Network. Contoso was aware that Azure Infrastructure Services supports VM-Generation ID which enabled them to create domain controllers on-premises and move them to an Azure Virtual Network, but decided that creating new domain controllers on the Azure Virtual Network would be the lower overhead option for the first phase of the project.

Given that the decision was made to put domain controllers on the Azure Virtual Network, Contoso needed to decide which disk types to use in Azure Infrastructure Services virtual machines to support the domain controller role. Contoso decided to adopt Microsoft's recommendations that the operating system be stored on an OS disk and all Active Directory-related files be stored on a Data Disk. This recommendation is based on the fact that OS disks enable write caching by default and Data Disks disable write caching by default, which is a core Active Directory assumption for disk behavior.

DESIGN DECISIONS:
-Contoso put two domain controllers on each Azure Virtual Network to support user authentication to applications
-Those domain controllers belonged to the same domain and forest as the on-premises production domain
-The new domain controllers were created on the Azure Virtual Network network
-Domain controllers used the disk type recommendations provided by Microsoft

The figure below provides a high level overview for application authentication. Notice that both the Internet-based client and the on-premises client connect to the web tier servers over the Internet. The web tier servers then send the credentials for authentication to the domain controllers on the Azure Virtual Network. The red arrows show the directory replication paths.

3.6.5.2 Read-Only Domain Controller

Contoso decided not to use read-only domain controllers because they knew that not all applications are compatible with read-only domain controllers and they did not want to incur the overhead of testing each application to determine if it was compatible with read-only domain controllers. This helped Contoso avoid the need to deploy both read-only and read-write domain controllers.

DESIGN DECISIONS:
-Contoso did not deploy read-only domain controllers

3.6.5.3 Domain Controller Locator

Contoso wanted to make sure that they minimized the impact of egress traffic due to replication traffic because they have to pay for Windows Azure egress traffic. The reason for this is that Contoso would have to pay for egress traffic. In order to achieve this goal, they decided to configure a subnet and site for each each Azure Virtual Network that they create in Azure Infrastructure Services. After creating sites, they configured site links with costs to insure that on-premises domain controllers see that Azure Virtual Network-located domain controllers see these sites as very high cost. They limited replication from the Azure Virtual Network-located domain controllers to weekdays only and created a replication policy that enforced this. Finally, Contoso decided to enable compression for Active Directory replication to decrease the amount of traffic (and egress cost) that they sent over the Azure Virtual Network.

DESIGN DECISIONS:
-Contoso created a subnet and site for each Azure Virtual Network
-High cost site links were set for links to domain controllers location on an Azure Virtual Network
-Replication policies were created that limited outbound replication to weekdays only
-Compression was enabled for Active Directory Replication

3.6.5.4 Domain, Forest and Global Catalog

Contoso had several options to choose from for deciding what type of domain and forest configuration to use for domain controllers and resources located on an Azure Virtual Network. They decided that the Azure security model was strong enough to enable them to put full read-write domain controllers that are members of the corporate on-premises domain and forest on the Azure Virtual Network. They decided to deploy domain controllers that are part of the same domain and forest as the on-premises network because it is easier to manage and deploy and enables the greatest level of application compatibility.

Contoso IT also decided to make the domain controllers located on the Azure Virtual Network global catalog servers. The reasons for this are that Contoso wanted to support Universal Groups, enable log on for their multiple-domain forest, and enable them to take advantage of universal group membership caching.

DESIGN DECISIONS:
-All domain controllers on the Azure Virtual Network were configured as global catalog servers
-All domain controllers on the Azure Virtual Network were read/write domain controllers
-All domain controllers on the Azure Virtual Network were members of the on-premises network forest and domain

3.6.5.5 Active Directory Name Resolution and Geo-Distribution

Contoso needed to decide how to handle name resolution for resources located on the Azure Virtual Network and resources located on-premises. While Contoso was aware that Azure Infrastructure Services provided some native DNS capabilities, they are not robust enough to support Active Directory's DNS requirements, such as support for SRV records. For this reason, Contoso decided to configure the domain controllers on AVNs to also be Active Directory integrated DNS servers.

DESIGN DECISIONS:
-All domain controllers located on the Azure Virtual Network were configured as AD-integrated DNS servers

3.6.5.6 Active Directory Federation Services (ADFS)

While Contoso didn't require federated access to applications hosted in Azure Infrastructure Services during the first phase of the hybrid cloud project, they did decide to use on-premises Active Directory accounts for logging in to Azure Infrastructure Services. In order to accomplish this, they federated their on-premises Active Directory infrastructure with Windows Azure Active Directory. Contoso plans to investigate application authentication federation when they move forward in the future to use Azure PaaS services. In addition, they plan on enabling same sign-on to Azure services by federating their on-premises directory with Azure Active Directory.

DESIGN DECISIONS:
-They will use ADFS for federating with Azure Active Directory to enable same sign-on

The final Active Directory authentication and authorization design can be summarized by the figure below. Contoso synchronized on-premises accounts with Azure directory services so that on-premises Active Directory accounts can be used to log on to the Azure portal. Active Directory Federation Services (ADFS) is used to to enable Azure Directory Services to forward credentials for verification with the Contoso Active Directory domain controllers. Organizational Units (OUs) are used to manage the configuration of both the web tier servers and the domain controllers in the Azure Virtual Network.

3.6.6 Backup Service and Disaster Recovery

In order to make sure that the company is able to recover from failures, Contoso IT needed to make design decisions around data backup and recovery for:

• Active Directory domain controllers on an Azure Virtual Network
• Domain controllers on an Azure Virtual Network

The following sections discuss the design decisions Contoso made in these areas and the rationale for those decisions.

3.6.6.1 Backup Services

Contoso hada well-established backup system in place for on-premises virtual machines and application data attached to those virtual machines. They decided that they will continue to use that system for on-premises resources. However, they will not use the same system to back up virtual machines and application data for resources running on Azure Infrastructure Services because they do not want to incur the egress traffic costs.

The pilot application's web tier hosts stateless services that are simple to restore, therefore there is little to gain by backing up the operating systems and there is no application data to back up.  In the next phase of the project, Contoso will consider using Windows Azure Backup Services. By using Windows Azure Backup Services they will be able to backup systems on Azure Infrastructure Services without incurring additional costs from standing up more virtual machines in the Azure Virtual Network to perform backup services.

DESIGN DECISIONS:
-Contoso did not backup the stateless machines that run the pilot application's web tier

3.6.6.2 Disaster Recovery

Though disaster recovery is one of the key usage scenarios they plan to evaluate in the future, they did not evaluate it during their pilot.

3.6.6.3 Operating System Update

Contoso needed to decide how they would update the Windows operating systems in the web tier virtual machines. They could have used the public Windows Update system or Windows Server Update Services. They have a WSUS implementation on-premises and have update policies in place for DMZ systems on-premises. They decided to update the pilot application's web tier servers with their existing, on-premises WSUS servers. This will not impact Contoso IT's network traffic costs because the traffic is inbound to the web tier servers over the Azure Virtual Network, rather than outbound, and they're only charged for outbound traffic. The updates will however, use available bandwidth over the site-to-site VPN connection and therefore updates of the web tier servers will be scheduled for off-hours.

DESIGN DECISIONS:
-Contoso used their on-premises WSUS system to update the web tier virtual machines
-They set update policies so that updates take place during off-hours

4.0 Summary

This article defined the design decisions, and rationale for the decisions, that a fictitious organization chose for its hybrid cloud infrastructure design. It selected options from the myriad of available options that are discussed in the Hybrid Cloud Infrastructure Design Considerations article. It selected the specific options that it did based on requirements that are defined in the Scenario Definition article. For organizations that have requirements and constraints similar to the organization discussed in the Scenario Definition article, the design decisions and rationale in this article can help decrease both your implementation time and risk of implementing a hybrid cloud solution.  If you could like to see how Contoso implemented the hybrid cloud solution, please see the Implementation guide. If you would like to learn about other cloud architectural solutions, please visit the the Cloud and Datacenter Solutions Hub.

5.0 Technologies Discussed in this Article

Windows Server 2012 DNS services
Active Directory Domain Services
Windows Azure Active Directory
Windows Azure Virtual Machines
Windows Azure Cloud Services
Windows Azure Storage
Windows Azure Storage Services
Windows Azure Recovery Service
Windows Azure Virtual Network 

6.0 Authors and Reviewers

Authors:
Thomas W. Shinder - Microsoft
Jim Dial - Microsoft

Reviewers: 
Yuri Diogenes - Microsoft
John Dawson - Microsoft

6.0 Change Log

To provide feedback on this article, leave a comment at the bottom of the article or send e-mail to SolutionsFeedback@Microsoft.com. To easily save, edit, or print your own copy of this article, please read How to Save, Edit, and Print TechNet Articles. When the contents of this article are updated, the version is incremented and changes are entered into the change log. The online version is the current version.

1.0  Introduction

This article is one of several articles that are included in the Cloud Services Foundation Reference Architecture (CSFRA) article set. An article set is a collection of interrelated articles that are intended to be read as a whole, like the chapters of a book. This article assumes that you have already read the Overview article for the set, which defines what a reference architecture is, what the Cloud Services Foundation is, and describes the relationship of this article to the other articles in the set. If you have not already read the Overview article, we recommend that you do so before you read this article.

This article defines the Cloud Services Foundation Reference Model (CSFRM). The CSFRM expands on the definition of the term cloud services foundation that is included in the Overview article of this article set.

1.1  Reference Models

Reference models share the following characteristics:

• They represent a problem domain
• They are often defined for problem domains that are not well understood or understood in a variety of different ways by different people, or that are sufficiently complex so that understanding them requires that the problem domain for which they’re created be decomposed into lower-level entities that promote common understanding
• They often consist of a diagram of entities, the relationships between the entities, and descriptive text that clearly defines each entity and relationship in the diagram
• They are typically vendor/product-agnostic and standards-agnostic to allow for various implementations that are based on them
• They provide common terminology in the problem domain for which they’re created
• They can serve as a foundation for designing and implementing solutions in the same problem domain for which they were created

The problem domain for the Cloud Services Foundation Reference Model (CSFRM) is cloud services foundation. Although the term is defined extensively in the Overview article of this article set, the short definition is:

The minimum amount of vendor-agnostic hardware and software technical capabilities and operational processes necessary to provide information technology (IT) services that exhibit cloud characteristics, or simply, cloud services.

It’s important to note that although the problem domain is the foundation for providing cloud services, it does not include cloud services. 

1.2  Usage

In addition to the attributes of reference models already listed, the CSFRM serves as a framework that can be used to help cloud services providers answer the following questions:

• What kinds of service level requirements should I define before I either design or implement a new cloud service or technical capabilities that support or enable cloud services?
• What kinds of operational processes do I require to operate a cloud service over its lifetime?
• What technical capabilities do I require to host, support, or manage cloud services?
• How will the services I provide be offered and presented to my consumers? 

2.0  Cloud Services Foundation Reference Model

Cloud Services Foundation Reference Model (Click any word in the diagram to be taken to that location in the article, or download an editable Visio diagram)

Many of the terms you see in the CSFRM diagram might look familiar to you. They might even cause you to wonder if and how this model is any different than what you have in your existing environment. Before you make any assumptions as to what anything in the CSFRM diagram represents, we recommend that you read through this article in its entirety for a clear explanation of everything that is represented in the diagram. After you read this entire article, the differences between what’s represented in the CSFRM diagram and what you have in your existing environment should be clear. When terms in the CSFRM diagram are used throughout the remainder of this article, they are capitalized to indicate that the words aren’t just used independently, but rather, that the words represent an entity from the diagram.

Note

Because the CSFRM diagram allows for many different interpretations, it should never be used without the detailed descriptions for each term, shape, and color that are represented in it. These descriptions are provided in the remainder of this article.

The CSFRM includes three types of entities:

1. Subdomains: The large blue and green boxes, some of which contain components
2. Components: The small boxes inside many of the subdomains
3. Relationships: The arrows between subdomains

Subdomains exist in the CSFRM to:

• Divide the cloud services foundation problem domain so that each subdomain can be defined separately.
• Enable a collection of components to be referred to collectively. For example, the components in the Infrastructure subdomain are Infrastructure components.
• Enable a relationship entity to represent the relationship between all of the components in a subdomain to the components of other subdomains. As a result, the relationships between subdomains then also collectively apply to the components that are contained in each subdomain. The relationships are represented by arrows in the model. The verbs by the arrows describe the relationship between the components in the subdomain that the arrow points from and the components in the subdomain that the arrow points to. Therefore, you could say that the Service Delivery subdomain components define the Service Operations subdomain components.

Note

The arrows pointing to and from the dashed line surrounding the Software, Platform, and Infrastructure subdomains represent the collective relationships between all of the components contained in the subdomains within the dashed line to the components in all of the other subdomains. Therefore, you could say that the Software, Platform, and Infrastructure subdomain components enable services for the Service Delivery subdomain components. You could also say that the Management and Support subdomain components manage and support the Software, Platform, and Infrastructure subdomain components.

Although each component represents a unique entity, all components in subdomains that are represented with the same color represent the same component type. Because there are two colors of subdomains that are represented in the CSFRM diagram, there are two types of components:

• Process: Green subdomains contain components that represent IT operational processes or service requirements or both. While many of the processes that are represented in the CSFRM are similar to those found in various information technology service management (ITSM) frameworks, there is no deliberate attempt to align the CSFRM processes to any particular framework, because different frameworks are used by different organizations. Therefore, while you might attempt to correlate the CSFRM processes to processes in the ITSM framework that you use, you likely will not find one-to-one mappings for all of the processes, nor will you necessarily find the definitions of the CSFRM processes in alignment with the definitions of the processes that are used in your framework of choice. Therefore, the processes that are represented in the CSFRM represent only the definition provided for each component in this article. Within an organization, the CSFRM processes can be manual, or automated, or be a combination of manual and automated. All processes and requirements for the components in process subdomains should be defined before designing or providing a cloud service to consumers. There are two process subdomains in the CSFRM, the Service Delivery subdomain and the Service Operations subdomain.
• Technical Capabilities: Blue subdomains contain technical capabilities components, which represent the functionality that is provided by hardware devices or software applications or both. Individual technical capabilities do not exhibit cloud characteristics, as defined in the Overview article of this article set on their own; a provider aggregates the functionality of several of them to provide cloud services. There are four technical capabilities subdomains in the CSFRM, Management and Support, Infrastructure, Platform, and Software. While capabilities such as physical structures or containers, cooling, and electricity are necessary to support technical capabilities, they are not represented in the CSFRM because they’re part of a separate problem domain, one that typically has a separate audience from the CSFRM.

The remaining sections of this article define each of the process and technical capabilities subdomains and their components in detail. In this article, the definitions for each subdomain and component are brief. The CSFRA article set, to which this article belongs, also includes detailed planning articles for many of the process and technical capabilities subdomains that are represented in the CSFRM.

Solution guidance that uses Microsoft products and technologies to implement the technical capabilities that are defined in this article and solution guidance for implementing various cloud services are provided separately from this article set and are available at the Cloud and Datacenter Solutions Hub.

2.1  Service Delivery

The components in this subdomain serve as the conduit for translating consumer requirements into cloud services and for the provider to manage the delivery of the services to the consumer requirements throughout the service lifecycle. This subdomain contains components that represent:

• Service level requirements that must be defined when:
  • Designing technical capabilities implementations where the provider owns or manages the technical capabilities that enable the service
  • Evaluating services provided by an external provider that owns or manages the technical capabilities that enable the service
• Processes that must be defined to ensure that a service meets its service level requirements throughout its lifecycle

It’s critical to define and measure cloud services requirements as specifically as possible to ensure ongoing customer satisfaction with the service. The components in this subdomain are defined in the following sections.

 Back to CSFRM diagram

2.1.1  Business Relationship Management

This component represents the following:

• Ongoing communication with consumers to determine if new services should be added or if existing services should be changed or deprecated. The list of services available to consumers is maintained in a service catalog, which is discussed further in the Consumer and Provider Portal component section of this article.
• The functional and service level requirements for services. Service level requirements are each represented in detail by other components in the Service Delivery subdomain.
• Defining and measuring customer satisfaction level targets for a service
• The plan for how the functional and service level requirements and customer satisfaction levels will be met. The plan typically includes clearly-defined requirements and consistently-applied processes from the Service Delivery and Service Operations subdomains, and often relies heavily on the Service Reporting component to monitor adherence to service level requirements. The Business Relationship Component has a close relationship to the Service Lifecycle Management and Service Level Management components. 

 CSFRM

2.1.2  Capacity Management

This component represents the following:

• The capacity requirements for scale and performance that the service must meet at daily, weekly, monthly, and annual intervals. Both peak and average requirements should be specified for each interval. Capacity requirements should also call out any specific peak times such as month-end, specific holidays, or year-end capacity requirements.
• The plan for how the requirements are to be met. The plan defines how technical capabilities that enable the service are scaled to meet capacity, whether the capabilities are owned and managed by the organization or an external provider or both. The plan applies the organization’s Request Fulfillment process and depends on the Fabric Management and Deployment and Provisioning components to provision new capacity.

 CSFRM

2.1.3  Availability and Continuity Management

This component represents the following:

• The availability level that the service adheres to after it’s available for use by consumers. This is often expressed in the percentage of time that the service is available during some timeframe, such as per month. The availability level should specify whether it includes planned downtime. The service provider might further define the requirement with “during certain hours or days”, or “under normal conditions” text.
• The plan for how the requirements are to be met. In defining the availability level that the service can adhere to and at what cost, the service provider typically evaluates its ability to provide:
  • Availability: This is generally thought of as the availability level of the service under normal conditions and includes the expected downtime from normal failures, such as a single-server failure.
  • Continuity: This is generally thought of as the availability level of the service under abnormal or disaster conditions, and includes the expected downtime from abnormal failures, such as an entire data center failure.

The cost to provide service levels under both normal and abnormal failure conditions can vary greatly. The provider generally iterates its design to reach an ultimate availability requirement that strikes the right balance between the availability level consumers desire and what they’re willing to pay for that availability level. To meet availability and continuity requirements for a service, the plan may utilize technical capabilities owned and managed by the organization or an external provider or both.

Note

The CSFRM recommends that availability metrics and availability and continuity plans be defined for each service that is provided in an organization. These plans are separate, but related to any business continuity plan that a provider might define for the organization as a whole.

 CSFRM

2.1.4  Information Security Management

This component represents the following:

• The confidentiality, integrity, and availability requirements of the data that is processed by the service. While each service might have unique requirements, there is generally a standard set of requirements that are used throughout the organization that are also applied.
• The plan for how these requirements are to be met by using the Access Management, Authentication, Authorization, and Data Protection components.

 CSFRM

2.1.5  Regulatory Policy and Compliance Management

This component represents the following:

• The regulatory compliance requirements for the service. The specific compliance requirements vary across countries and industries and might apply to every service in an organization, specific services in an organization, or specific data in individual services. Examples of regulatory policies that might influence the design or selection of a public service are the Health Insurance Portability and Accountability Act (HIPAA), Payment Card Industry (PCI), Sarbanes-Oxley (SOX), European Union Directive 95/46/EC, International Convergence of Capital Measurement and Capital Standards (Basel II), Data Protection Act of 1998, and others.
• The plan for how these requirements are to be met by using the Access Management, Authorization, Authentication, Data Protection, and Service Reporting components.

 CSFRM

2.1.6  Financial Management

This component represents the following:

• Definitions of the cost to provide the service and what consumers pay for the service. The cost to provide the service can’t be finalized without incorporating the cost to meet requirements from the following components:
• Capacity Management
• Availability and Continuity Management
• Information Security Management
• Regulatory and Compliance Management
• The plan for how these requirements are to be met. The service provider must continually evaluate its costs to provide the service and adjust the cost of the service to its consumers as appropriate. A key goal for most service providers is to minimize their cost of delivering a service to their consumers. The Usage and Billing component is a key enabler for recovering costs to provide the service, while efficient Service Operations processes and the Process Automation component are key enablers for decreasing the cost of providing a service.

 CSFRM

2.1.7  Service Level Management

This component represents the following:

• Requirements for many of the processes that are defined in the Service Operations subdomain components in this article.
• Requirements for many of the technical capabilities that are defined in the Management and Support subdomain components in this article.
• The plan for how these requirements are to be met. While the processes and technical capability requirements might differ across services, they generally do not dramatically differ, because most organizations define a standard, and then try to adhere to it as much as possible.

This component is a key enabler for customer satisfaction and results in the service level agreement (SLA) for the service, which is created from the outcomes of many of the components in this subdomain. This component has a close relationship to both the Business Relationship Management component and all Service Operations components. 

 CSFRM

2.1.8 Service Lifecycle Management

This component represents the overall lifecycle processes of each service from requirements definition through design and implementation, operations with the Service Operations components, and eventual retirement. This component has a close relationship to both the Business Relationship Management component and all Service Operations components. 

 CSFRM

2.2  Service Operations

 The components in this subdomain represent the processes that are applied to each service to ensure that it continuously meets the requirements that are defined by the components in the Service Delivery subdomain. Although many organizations define each of these components as standardized processes, the specific application of the processes often varies across services. The Management and Support components support the components of this subdomain.

The following sections provide high-level detail about each of the process components in this subdomain. While many of the processes might be similar to the processes used to operate services without cloud characteristics in an organization, there are a few notable differences when operating services with cloud characteristics:

• Processes are typically highly automated to minimize human error and labor
• Consistent application of the processes is critical to meeting service level requirements. While this might state the obvious, many organizations that don’t provide cloud services to their consumers might not be as diligent in their operational processes as organizations that do provide cloud services.

 CSFRM

2.2.1 Request Fulfillment

This component defines how consumer requests for service are fulfilled. The process defines requirements for how the Consumer Portal, Deployment and Provisioning, and Fabric Management components are used to fulfill requests, but also defines how Infrastructure, Platform, and Software technical capabilities will be acquired from external vendors. This process is a key enabler to meeting the requirements represented by the Capacity Management component.

 CSFRM

2.2.2 Access Management

This component defines how the requirements for the service from the Information Security Management component are applied and complied with. The Authentication, Authorization, and Data Protection components support the Access Management component.

 CSFRM

2.2.3 Asset and Configuration Management

This component defines:

• How software and hardware assets that provide technical capabilities are managed by the organization, from asset acquisition to asset retirement.
• The assets that enable services and the configuration items (CIs) for each asset that are managed by the Change Management component for each service. Changes to some CIs of an asset may impact the SLA for the service, while others do not. Only the CIs that impact the SLA of the service are managed by the Change Management component.
• The value of the desired state for each CI of each service, how the current value of the CI will be evaluated against the desired state, and what the mitigation process is when the current value of the CI is not the desired state value.

The outcomes of this component define requirements for the Service Management and Configuration Management components and are utilized by the Change Management component. 

 CSFRM

2.2.4 Systems Administration

This component represents the daily, weekly, monthly, and as-needed tasks that are required for the service to meet SLAs throughout its lifecycle. As many tasks as possible are often automated with the Process Automation component.

 CSFRM

2.2.5  Change Management

This component represents the process that enables the provider to determine and weigh the benefits of making a change to the proposed state of CIs of a service against the risk of not being able to meet the SLA metrics for the service. The change requests for which the benefits outweigh the risks are approved. This process is supported by the Service Level Management and Configuration Management components, but utilizes the CIs defined by the Configuration Management component.

 CSFRM

2.2.6  Knowledge Management

This component represents knowledge about the service that is to be gathered, analyzed, stored, and shared throughout the organization. This knowledge is often maintained by the Service Management component and used by the Incident and Problem Management component.

 CSFRM

2.2.7  Release and Deployment Management

This component represents the process that defines how change requests that are approved by the Change Management component are tested and deployed to production with the Deployment and Provisioning component, while still enabling the service to meet its SLAs.

 CSFRM

2.2.8  Incident and Problem Management

This component represents the process for:

• Resolving incidents: Incidents are disruptive or potentially disruptive events to a service. The goal of this process is to resolve the incident with maximum speed and minimum service disruption, but identifying the underlying root cause of the incident is not a goal of this process.
• Identifying and resolving problems: Problems are the underlying root cause of one or more incidents. Problems are often more difficult to identify and resolve than incidents are, but their resolution can prevent future incidents.

The Service Management component is used to help identify both resolutions to incidents, and recurring incidents that require problem resolution. The individual who resolves the incident might be a member of the service’s support staff, or might be a consumer that used the Consumer Portal component to resolve the issue themselves. The individual that resolves problems is often a subject matter expert or a team of subject matter experts that have a deep understanding of the technical capabilities that enable the service.

 CSFRM

2.3  Management and Support

The technical capabilities components within this subdomain manage and support on-premises technical capabilities that host, manage, or support private, externally-consumed public, or hybrid cloud services. The requirements that these capabilities meet are defined by both the Service Delivery and Service Operations components in the environment, but the components also provide data to the Consumer and Provider Portal component that enables consumers to monitor whether the provider adhered to the SLA requirements defined by the Service Level Management component. When you select technologies to implement these components, keep in mind that the functionality of multiple components may be provided by a single technology or that multiple technologies may be required to provide a single component, or that multiple technologies may provide the same component, but in different ways. 

 CSFRM

2.3.1 Consumer and Provider Portal

This component is the consumers' interface to the services that are available in their environment. It includes:

• The service catalog, which lists the available services, the SLAs that the services offer, and the consumption cost for the services. New services aren’t listed in the service catalog until they meet SLA requirements that are defined by the Service Level Management component. An organization may consume a service from an external provider that manages the technical capabilities that enable the service. The consuming organization may aggregate the service capabilities with technical capabilities that it manages and then provide the service to consumers within its own organization. This is often referred to as a hybrid deployment model for a service. Alternatively, an organization may provide a service to consumers within its own organization that is enabled solely with technical capabilities managed by the organization. This is often referred to as a private cloud deployment model for a service.
• A self-service interface to the Deployment and Provisioning and Fabric Management components that allow consumers to provision new capacity as required.
• A reporting interface that exposes information from both the Usage and Billing and Service Monitoring components.
• Incident and problem resolution data from the Service Management component so that consumers can resolve issues themselves if they choose.

This component also provides a portal for the provider to manage the services that they provide to their consumers.

 CSFRM

2.3.2 Usage and Billing

This component collects usage data and presents it in the Consumer and Provider Portal component. This enables consumers to understand the number of consumption units that they used for a service over a given time period and what that usage cost. Many enterprise IT organizations might use this data to provide “show back” reports instead of actual billing to consumers, so they can understand how many of the organizations’ resources they consume each month. Hosting service provider (HSP) organizations use this data for customer billing.

 CSFRM

2.3.3 Service Reporting

This component consumes Service Monitoring data and produces reports that describe the actual service level metric values exhibited by a service over regular time intervals. The report data can be compared to SLAs to determine whether the service met its SLAs during the reporting interval that is specified in the SLA. The data from this component is provided to consumers through the Consumer and Provider Portal component. This component is a primary enabler for the Service Level Management and Business Relationship Management components.

 CSFRM

2.3.4 Service Monitoring

This component monitors service levels of all technical capabilities that are used to provide each cloud service. The Service Reporting component consumes the data from this component. Optimally, the Service Monitoring Component is able to integrate with the Service Management component so that it can auto-generate incidents based on defined criteria.

 CSFRM

2.3.5 Service Management

This component supports most of the Service Operations components and integrates data from the following Management and Support components:

• Incident and Problem Management
• Configuration Management
• Service Monitoring
• Service Reporting

The data integrated by the Service Management component is typically exposed through the Consumer Portal component so that various individuals can view it. 

 CSFRM

2.3.6 Configuration Management

This component is the definitive store of both all of the CIs in the environment, and their relationships. The component might be a physical or logical store that correlates data across several different configuration management databases (CMDBs), each of which manages a subset of the CI data. Ideally, the component can also monitor CIs within the environment to identify, report, and repair CIs when their values change from their defined desired states. This component directly supports the Change Management and Asset and Configuration Management components.

 CSFRM

2.3.7 Authorization

This component denies or grants access to technical capabilities components based on access controls that are assigned to entities that have been authenticated by the Authentication component. As a result, this component has a relationship to the Authentication component that enables it to confirm whether the entity that requested authorization to a resource has been authenticated. Federation of the Authorization component across different cloud service providers is particularly valuable to provide the most seamless experience to end-users of services, by minimizing the amount of authorization strategies that must be synchronized across cloud services providers. This component directly supports the Access Management and Information Security Management components.

 CSFRM

2.3.8 Authentication

This component validates that an entity is who or what it claims to be based on some form of proof. In its simplest form,proof can be a user name and password. Authentication could also rely on digital certificates for proof. The entity that’s authenticated might be a human user or a software application that has to interact with another software application. Federation of the Authentication component across different cloud service providers is particularly valuable to provide the most seamless experience to end-users of services, by minimizing the amount of times that they must authenticate to various services. 

 CSFRM

2.3.9 Directory

This component stores entities and their attributes in a directory. The entities could be of a variety of types such as computers or user identities. The component also provides a mechanism to query the directory for the attributes of entities. For example, the Authentication component might query a directory for the user name and password attributes of entities that attempt to authenticate to it.

 CSFRM

2.3.10 Data Protection

This component protects data in the event of data corruption or loss or underlying storage capability failures, or any combination of these events. It directly supports data retention policies that support the Information Security Management, Availability and Continuity Management, and Regulatory Policy and Compliance Management components.

 CSFRM

2.3.11 Deployment and Provisioning

This component executes the outcomes of the Release and Deployment Management component for each service to provision new units of capacity on an Infrastructure fabric. As a result, this component often interacts with the Fabric Management component to provision new Infrastructure components to support service capacity needs. This component directly supports the Capacity Management and Request Fulfillment components.

 CSFRM

2.3.12 Process Automation

This component coordinates automated processes across multiple Management and Support and Infrastructure components. It ensures that processes are completed in accordance with their defined tasks. This component directly supports automation of many of the Service Delivery and Service Operations components.

 CSFRM

2.3.13 Fabric Management

This component controls all Infrastructure components through the Virtualization component. The Infrastructure components aren’t referred to as a “fabric” unless they’re managed with a Fabric Management component. This component directly supports the Capacity Management and Request Fulfillment components and directly interacts with the Network Support component to ensure network accessibility of Infrastructure components. The Deployment and Provisioning component often interacts with this component to support service capacity needs.

 CSFRM

2.3.14 Network Support

This component includes functionality that enables the use of network protocols that are used by Infrastructure component to communicate with each other and other devices. It typically includes functionality such as dynamic host configuration protocol for internet protocol (IP) address assignment and management, domain name system for IP name and address resolution, and pre-boot execution environment to enable a network interface-based boot of the Compute component without direct-attached storage (DAS) or operating system. This capability directly supports the Infrastructure components.

 CSFRM

2.4  Infrastructure

The components within this subdomain represent the technical capabilities that you manage that are required to host on-premises Platform, Software, and Management and Support technical capabilities components. You might use these components to enable IaaS services that you provide to consumers in the public, private, community, or hybrid deployment models¹. You may also consume IaaS services provided by an external provider that manages the infrastructure components for IaaS services. The requirements that these components meet are driven by all subdomains within the CSFRM, but are generally heavily-standardized to facilitate both automation in the environment, and to optimize volume purchases of hardware and software. 

 CSFRM

2.4.1 Virtualization

This component abstracts some, but rarely all, of the functionality of the individual Compute, Network, and Storage components. For example, a physical server has basic input/output system (BIOS) settings. In modern servers, one of the available BIOS settings enables you to turn on or off hardware-assisted virtualization functionality, which typically must be turned on for the server to run any modern hypervisor software. The hypervisor is a key enabler to provide a virtual machine service. Even if the hardware-assisted virtualization setting could be virtualized, it likely wouldn’t be, because if the setting weren’t enabled, the virtual machine service couldn’t even be provided. As a result, the functionality that is virtualized is only the functionality that is necessary to provide the requested service to the consumer. Even though virtualization is not required to provide cloud services, it is a key enabler for doing so, and therefore, is represented as a component in the CSFRM.

 CSFRM

2.4.2 Compute

This component represents physical servers, which include resources such as processors, graphics processing units (GPUs), random access memory (RAM), network interfaces, and the storage necessary to host hypervisor software for the physical server.

 CSFRM

2.4.3 Network

This component represents the physical network switches, routers, firewalls, and cabling. It also represents logical networking constructs including virtual local area networks (LANs), access control lists, quality of service, and network interfaces defined in converged network architectures.

 CSFRM

2.4.4 Storage

This component represents physical storage that is accessed by Compute devices via some networking technology. Types of data that are often stored on this component are virtual machine hard disk files or consumer data or both.

Although the Infrastructure and Management and Support components that are necessary to provide services in the infrastructure as a service (IaaS) service model¹ exist in the CSFRM, they are not represented solely to enable IaaS services. They’re represented because they’re necessary to enable any type of cloud service.

 CSFRM

2.5  Platform

Platform components are aggregated with Infrastructure and Management and Support components to provide platform as a service (PaaS) services¹. This subdomain does not include any technical capabilities components because Platform components are not critical to provide services in every service model, whereas the other technical capabilities components that are represented in the CSFRM are. Platform technical capabilities components are however, critical to providing platform services. If you provide platform services in your environment, either with platform technical capabilities that you manage or with platform technical capabilities managed by an external provider, you can add Platform components to the CSFRM, as appropriate. Examples of platform services that you might provide in your organization are data, media, and service bus. Platform components are sometimes provided as services that are consumed by Software capabilities, but they may also be consumed directly by end-users.

 CSFRM

2.6  Software

Software components are aggregated with Infrastructure, Management and Support, and sometimes Platform components to provide software as a service (SaaS) services¹. This subdomain does not include any components because Software components are not critical to providing services in every service model, whereas the other components that are represented in the model are. Software technical capabilities components are however, critical to providing software services. If you provide software services in your environment, either with software technical capabilities that you manage or with software technical capabilities managed by an external provider, you can add Software components to the CSFRM, as appropriate. Examples of software services that you might provide in your organization are email, calendaring, customer relationship management, enterprise resource planning, unified communications, and collaboration. Software services are usually consumed by end users.

 CSFRM

3.0  Summary

This article introduced the Cloud Services Foundation Reference Model, which defined common terminology for subdomains and components within the cloud services foundation problem domain. You might have some or all of the components in your environment today, but you might not use them in a manner that enables you to provide services that exhibit cloud characteristics. Providing cloud services requires more than common terminology though. A guiding set of principles, concepts, and patterns that can be applied to the implementation of a cloud services foundation is also required. This information is provided in the Principles, Patterns, and Concepts article in this article set. We recommend that you read that article next. You may also want to return to the Cloud Services Foundation Reference Architecture - Overview article or obtain additional architectural and solutions content by visiting the Cloud and Datacenter Solutions Hub.
 

¹Refers to terminology from the NIST Definition of Cloud Computing. Terminology from this definition is used throughout this article, but is not defined further with this article. 

4.0 Change Log