Storage Spaces Direct in Windows Server 2016 Technical Preview

 

Updated: February 4, 2016

Applies To: Windows Server Technical Preview

Windows Server 2016 Technical Preview introduces Storage Spaces Direct, which enables building highly available (HA) storage systems with local storage. This is a significant step forward in Microsoft Windows Server software-defined storage (SDS) as it simplifies the deployment and management of SDS systems and also unlocks use of new classes of disk devices, such as SATA and NVMe disk devices, that were previously not possible with clustered Storage Spaces with shared disks.

With Windows Server 2016 Technical Preview Storage Spaces Direct, you can now build HA Storage Systems using storage nodes with only local storage, which is either disk devices that are internal to each storage node (Figure 1), or disk devices in JBODs where each JBOD is only connected to a single storage node (Figure 2). This completely eliminates the need for a shared SAS fabric and its complexities, but also enables using devices such as SATA disks, which can help further reduce cost or NVMe devices to improve performance.

FIGURE 1: Storage Spaces with internal disks

FIGURE 2: Storage Spaces Direct with JBODs

Storage Spaces Direct is an evolution of Storage Spaces, which means that it is an extension of the existing software defined storage stack for Windows Server. Storage Spaces Direct leverages SMB3 for all intra-node (aka east-west) communication, including SMB Direct and SMB Multichannel, for low latency and high throughput storage.

Overview

Storage Spaces Direct seamlessly integrates with the features you know today that make up the Windows Server software defined storage stack, including Scale-Out File Server, Clustered Shared Volume File System (CSVFS), Storage Spaces and Failover Clustering. Figure 3 below illustrates the “Storage Spaces Direct” stack:

FIGURE 3: Storage Spaces Direct stack

The Storage Spaces Direct stack includes the following, starting from the bottom:

Networking hardware Storage Spaces Direct relies on a network to communicate between hosts. For production deployments, it is required to have an RDMA-capable NIC (or a pair of NIC ports).

Storage hardware: The storage system consisting of a minimum of four storage nodes with local storage. Each storage node can have internal disks, or disks in an external SAS connected JBOD enclosure. The disk devices can be SATA disks, NVMe disks or SAS disks.

Software Storage Bus: The Software Storage Bus is the Storage Spaces Direct specific software component that spans all the storage nodes and brings together the local storage in each node, so all disks are visible to the Storage Spaces layer above. For more information about Software Storage Bus, see Storage Spaces Direct - Under the hood with the Software Storage Bus.

Storage Pool: The storage pool spans local storage across all the nodes.

Storage Spaces: Storage Spaces (aka virtual disks) provide resiliency to disk or node failures as data copies are stored on different storage nodes.

Resilient File System (ReFS) ReFS provides the file system in which the Hyper-V VM files are stored. ReFS is a premier file system for virtualized deployments and includes optimizations for Storage Spaces such as error detection and automatic correction. In addition, ReFS provides accelerations for VHD(X) operations such as fixed VHD(X) creation, dynamic VHD(X) growth, and VHD(X) merge.

Clustered Shared Volumes: CSVFS layers above ReFS to bring all the mounted volumes into a single namespace accessible through any node.

Scale-Out File Server This is the top layer of the storage stack that provides remote access to the storage system using the SMB3 access protocol. The Scale-Out File Server (SOFS) layer is only needed in disaggregated configurations (where the Storage Spaces Direct system is dedicated to providing storage services), and is not implemented in hyper-converged configurations (where the virtual machines are hosted on the same cluster as the Storage Spaces Direct system).

Use Cases

Storage Spaces Direct can be deployed either for primary storage of Hyper-V VM file, or, for secondary storage for Hyper-V Replica virtual machine files. In addition, the deployment can be for backup or for archive of virtual machine files.

Disaggregated and Hyper-converged Configurations

There are two targeted deployment scenarios for Windows Server 2016 Technical Preview Storage Spaces Direct. Both cases provide storage for Hyper-V, specifically focusing on Hyper-V IaaS (Infrastructure as a Service) for Service Providers and Enterprises.

The disaggregated deployment scenario has the Hyper-V servers (compute component) in a separate cluster from the Storage Spaces Direct servers (storage component). Virtual machines are configured to store their files on the Scale-Out File Server which is accessed through the network using the SMB3 protocol. This allows for scaling Hyper-V clusters (compute) and Scale Out File Server cluster (storage) independently. For example, the compute nodes are nearing capacity for the number of VMs that they can host but the storage has excess capacity (both disk and IOPS), more compute nodes can be added without adding additional storage nodes. Figure 4 illustrates the disaggregated deployment scenario.

FIGURE 4: Disaggregated deployment of Storage Spaces Direct

The hyper-converged deployment scenario has the Hyper-V (compute) and Storage Spaces Direct (storage) components on the same cluster. Virtual machine's files are stored on the local CSVs and does not implement a Scale-Out File Server. This allows for scaling Hyper-V compute clusters and storage together and removes requirement of configuring file server access and permissions. Once Storage Spaces Direct is configured and the CSV volumes are available, configuring and provisioning Hyper-V is the same process and uses the same tools that you would use with any other Hyper-V deployment on a failover cluster. Figure 5 illustrates the hyper-converged deployment scenario.

FIGURE 5: Hyperconverged – same cluster configured for Storage Spaces Direct and the hosting of virtual machines

Hardware requirements

We are working with our hardware partners to define and validate specific hardware configurations, including SAS HBA, SATA SSD and HDD, RDMA enabled network adapters etc. to make sure of a good user experience.

Use PowerShell to deploy and manage Storage Spaces Direct. Do not use Server Manager or Failover Cluster Manager to manage Storage Spaces Direct.

To evaluate Storage Spaces Direct in Windows Server 2016 Technical Preview, the simplest deployment is to use at least four generation 2 Hyper-V virtual machines with at least four data disks per virtual machine. To test Storage Spaces Direct in Windows Server 2016 Technical Preview, see Testing Storage Spaces Direct using Windows Server 2016 virtual machines.

For more information about hardware options, see Hardware options for evaluating Storage Spaces Direct in Technical Preview 4

Note

Storage Spaces Direct does not support disks connected via multiple paths, and the Microsoft Multipath MPIO software stack.

Configurations using different disk types

Storage Spaces Direct in Windows Server 2016 Technical Preview supports use of locally connected disks such as SAS HBA and SATA connected HDD, SATA SSD, and NVMe. (see the Hardware requirements section for guidance on selecting the specific devices). These devices can be used in different configurations to allow flexibility to choose based on performance and cost. The Deploy Storage Spaces Direct section of this guide has the steps to configure a system that uses a mix of HDD and SDD, with the SDD disks configured for use as system cache and Journaling. . You can also deploy a configuration that has a mix of all high performance flash storage like NVMe and SATA SSD disks. The steps to deploy this “all-flash” configuration are identical with the example outlined below, however, the Windows PowerShell cmdlets for steps 4-6 are slightly different. The PowerShell cmdlet examples for all-flash deployments are located in the Windows PowerShell example for all-flash configurations section.

Installing and configuring Storage Spaces Direct

This section includes instructions to install and configure Storage Spaces Direct in Windows Server 2016 Technical Preview and includes the following sections:

• Deploy the operating system

• Deploy the network

• Deploy Storage Spaces Direct

Important

This preview release should not be used in production environments.

Deploy the operating system

Complete the following steps to deploy the operating system:

1. Install the operating system on each node.

   On each node, install Windows Server Technical Preview 3 using the media provided or a download from the Microsoft’s website.

2. Join each node to the same Windows Active Directory domain.

Deploy the Network

In order to deploy Storage Spaces Direct, the Hyper-V switch must be deployed with RDMA-enabled host virtual NICs. The administrator may decide whether to use a single physical NIC port or team two or more physical NIC ports. Complete the following steps to deploy the network:

Note

Skip this Deploy the Network section, if you are testing Storage Spaces Direct inside of virtual machines. RDMA is not available for networking inside a virtual machine.

Note

This “Deploy the Network” section is required for both the disaggregated and hyper-converged deployment scenarios for any deployment that uses RDMA enabled networking.

Note

For Windows Server 2016 Technical Preview, there are multiple vendors supporting these RDMA network capabilities. Currently, NICs from Mellanox (CX3-Pro), Chelsio (T5), and Avago/Emulex (Skyhawk) have claimed support of these network capabilities. Other vendors are also working on supporting these networking capabilities, however, not all vendor solutions have been tested by Microsoft.

• Step 1. Enable the Hyper-V role

• Step 2. Create the vSwitch

Step 1. Enable the Hyper-V role

After you have installed the operating system, you must enable the Hyper-V role to install the new RDMA capable virtual switch (vSwitch). You can enable the role using PowerShell as follows:

Install-WindowsFeature –Name Hyper-V -IncludeManagementTools –Restart

Note

Installing the Hyper-V role requires a restart of the system.

For step-by-step instructions about installing the Hyper-V role, see Install Hyper-V and create a virtual machine

Step 2. Create the vSwitch

If a single NIC-port will be used on the host, create the vSwitch using that NIC-port:

New-VMSwitch -Name S2DSwitch -NetAdapterName "NIC 1"

If multiple NICs will be teamed for this host, create the switch with switch-embedded teaming (SET) enabled. For more information, see Remote Direct Memory Access (RDMA) and Switch Embedded Teaming (SET).

New-VMSwitch -Name S2DSwitch -NetAdapterName "NIC 1",”NIC 2”

A single NIC-port may be sufficient depending on the workloads and the NIC speed. For example, a single 40Gb/s NIC port will perform faster than two 10 Gb/s NIC ports. However, some customers are more comfortable with two paths, two cables, and the ability to failover between them. For those customers, creating a switch with an embedded team meets their needs.

After you have created the virtual machine switch and configured two NIC-ports, the next step is to add virtual network adapters and enable RDMA on them:

Add-VMNetworkAdapter -SwitchName S2DSwitch -Name SMB1 -ManagementOS
Add-VMNetworkAdapter -SwitchName S2DSwitch -Name SMB2 -ManagementOS
Enable-NetAdapterRDMA "vEthernet (SMB1)","vEthernet (SMB2)"

Confirm that the adapters are created and enabled for RDMA. The following PowerShell cmdlet should show that SMB1 and SMB2 are present and RDMA-enabled:

 Get-NetAdapterRdma

Name                    InterfaceDescription                     Enabled
----                    --------------------                     -------
vEthernet (SD2Switch)   Hyper-V Virtual Ethernet Adapter #8      False
SMB1 (SD2Switch)        Hyper-V Virtual Ethernet Adapter #7      True
SMB2 (SD2Switch)        Hyper-V Virtual Ethernet Adapter #7      True 
SLOT 3                  Mellanox ConnectX-3 Pro Ethernet Adapter True
SLOT 3 2                Mellanox ConnectX-3 Pro Ethernet Adap... True

Next, configure each virtual network interface with the appropriate IP addresses

Deploy Storage Spaces Direct

Deploying Storage Spaces Direct in Windows Server 2016 Technical Preview includes the following steps:

• Step 1. Install the Windows Server roles and features

• Step 2. Validate

• Step 3. Create a cluster

• Step 4. Enable Storage Spaces Direct

• Step 5. Create storage pools

• Step 6. Create storage tiers

• Step 7. Create virtual disks

• Step 8. Create a file server

• Step 9. Create file shares

Step 1. Install the Windows Server roles and features

Use the following PowerShell command on each node to install the File Server role and the Failover Clustering feature.

Install-WindowsFeature –Name File-Services, Failover-Clustering –IncludeManagementTools

Step 2. Validate

Run cluster validation to catch common configuration problems and verify that the servers are ready to create a cluster with Direct Storage.

Note

The “-Include” switch needs to be used to specify the “Storage Spaces Direct” test category. If this is not specified the “Storage” category of tests for shared storage will run.

Use the following PowerShell command to validate a set of servers for use as a Storage Spaces Direct cluster.

#Validate cluster
Test-Cluster –Node <NodeName1, NodeName2, NodeName3, NodeName4> –Include “Storage Spaces Direct”,Inventory,Network,”System Configuration”

Note

You can run Validate after the cluster is created using the same command as specified above and adding “Cluster Configuration” to the included test categories. “Cluster Configuration” cannot be run prior to cluster creation since no cluster is available to get configuration information from.

Step 3. Create a cluster

Create a cluster that includes the file servers.

Note

Storage Spaces Direct is supported on Nano Server. For more information about configuring Failover Clustering on a Nano Server, see Getting Started with Nano Server.

Note

Disks with existing partitions are not eligible and will not be claimed by the software storage bus. It is recommended that you clear such disks before creating the cluster or enabling the software storage bus. You can use the Clear-Disk PowerShell command to clean the disks, which removes all data from the disks.

Use the following PowerShell command to create a cluster with four nodes.

#Create cluster
New-Cluster –Name <ClusterName> –Node <NodeName1,NodeName2,NodeName3,NodeName4> –NoStorage

If you are using static IP addresses for the cluster nodes, modify the command to reflect the static IP address for the cluster as follows:

#Create cluster
New-Cluster –Name <ClusterName> –Node <NodeName1,NodeName2,NodeName3,NodeName4> –NoStorage –StaticAddress <X.X.X.X>

Note

When creating the cluster you will get a warning that states - “There were issues while creating the clustered role that may prevent it from starting. For more information view the report file below.” You can safely ignore this warning.

Once the cluster has been formed for disaggregated deployments, ensure that the appropriate cluster network interfaces are enabled for client access. This is not required for hyper-converged deployments.

Any cluster network that will be used to access the storage system should be set to “ClusterAndClient”. The following is an example showing cluster networks and setting one of them from Cluster to ClusterAndClient”:

PS C:\Windows\system32> Get-ClusterNetwork

Name              State Metric             Role
----              ----- ------             ----
Cluster Network 1    Up  30384          Cluster
Cluster Network 2    Up  30385          Cluster
Cluster Network 3    Up  70400 ClusterAndClient


PS C:\Windows\system32> (get-ClusterNetwork "Cluster Network 1").Role = "ClusterAndClient"
PS C:\Windows\system32> Get-ClusterNetwork

Name              State Metric             Role
----              ----- ------             ----
Cluster Network 1    Up  70384 ClusterAndClient
Cluster Network 2    Up  30385          Cluster
Cluster Network 3    Up  70400 ClusterAndClient

Step 4. Enable Storage Spaces Direct

After creating the cluster, use the following PowerShell command to set the cluster properties to enable the software storage bus. . The commands used depends on the disk devices available in the system.

For SATA SSD + SATA HDD configuration, you enable Storage Spaces Direct as follows:

# Enable Storage Spaces Direct
Enable-ClusterStorageSpacesDirect

For NVMe SSD + SATA SSD configuration, you can enable Storage Spces Direct as follows:

# Enable Storage Spaces Direct
Enable-ClusterS2D –S2DCacheDevice NVMe

For configurations where it’s all NVMe or all SATA SSD (not both), you can enable Storage Spces Direct as follows:

# Enable Storage Spaces Direct for all flash of a single type, disabling the S2D Cache Mode
Enable-ClusterS2D -S2DCacheMode Disabled

In a SATA SSD + SATA HDD storage configuration, Storage Spaces Direct uses the SATA SSD as performance devices and the SATA HDD as capacity devices. The performance devices are used for read/write caching and metadata.

In a NVMe SSD + SATA SSD storage configuration, Storage Spaces Direct uses the NVMe SSD as performance devices and the SATA SSD as capacity devices. The performance devices are used for write caching and metadata.

Step 5. Create storage pools

Use the following PowerShell command to create a storage pool:

#Create storage pool
New-StoragePool  -StorageSubSystemName <FQDN of the subsystem> -FriendlyName <StoragePoolName> -ProvisioningTypeDefault Fixed -ResiliencySettingNameDefault Mirror -PhysicalDisk (Get-StorageSubSystem  -Name <FQDN of the subsystem> | Get-PhysicalDisk)
Get-StoragePool <StoragePoolName> | Get-PhysicalDisk |? MediaType -eq SSD | Set-PhysicalDisk -Usage Journal

Use the following PowerShell command to set journal devices in a SATA SSD + SATA HDD storage configuration. In this configuration all the performance devices (SATA SSD) must be set as journal devices.

# Set SSD usage to journal
Get-StoragePool <StoragePoolName> | Get-PhysicalDisk |? MediaType -eq SSD | Set-PhysicalDisk -Usage Journal

Use the following PowerShell command to set journal devices in a NVMe SSD + SATA SSD storage configuration. In this configuration all the performance devices (NVMe SSD) must be set as journal devices:

#Set NVMe usage to journal
Get-StoragePool <StoragePoolName> | Get-PhysicalDisk |? MediaType -eq SSD –and BusType –eq NVMe | Set-PhysicalDisk -Usage Journal

In addition to forming storage pools, it also sets the default resiliency to mirror and disables the Write Back Cache, both of which govern the subsequent creation of virtual disks. Note that in Storage Spaces Direct you should not set EnclosureAwareness, for more information see the Storage Spaces Fault Tolerance section.

Step 6. Create storage tiers

Use the following PowerShell commands to create a storage tiers for SSD + HDD configurations:

New-StorageTier -StoragePoolFriendlyName <StoragePoolName> -FriendlyName <MirrorTierName> -MediaType HDD -ResiliencySettingName Mirror -NumberOfColumns 4 -PhysicalDiskRedundancy 2
New-StorageTier -StoragePoolFriendlyName <StoragePoolName> -FriendlyName <ParityTierName> -MediaType HDD -ResiliencySettingName Parity -NumberOfColumns 7 -PhysicalDiskRedundancy 2

Note

For NVMe + SSD configuration, see Windows PowerShell example for all-flash configurations later in this document.

Step 7. Create virtual disks

After creating the storage pool, you must create the virtual disks. Mixed resiliency tiering, which is new in Technical Preview 4, enables creating virtual disks that consists partly of mirror storage and partly of parity (erasure coding) storage. Combined with enhancements in ReFS, this enables better storage efficiency through use of parity (erasure coding), while still providing good IO performance through use of mirroring of hot data.

The following PowerShell command creates a virtual disk with both mirror and parity resiliency:

Note

Technical Preview 4 introduces “Multi-Resiliency Virtual Disks”, which allows one virtual disk to include two different resiliency types. For instance, the example below sets the virtual disk to have 3-way mirror and parity. ReFS will then do writes to the mirrored blocks of the volume and as the mirrored section becomes full, they will be converted in the background to parity and parity blocks will be reallocated to the mirror section. This allows for optimizing writes to the mirror for performance and in the background converting to parity when needed to optimize physical disk usage.

#Get the storage tier definitions
$mt = Get-StorageTier <MirrorTierName>
$pt = Get-StorageTier <ParityTierName>
#Create mirror + parity virtual disks
New-Volume -StoragePoolFriendlyName <StoragePoolName> -FriendlyName <VirtualDiskName> -FileSystem CSVFS_ReFS -StorageTiers $mt,$pt -StorageTierSizes <Size of mirror in size units>, <Size of parity in size units>

Important

In Technical Preview 4, parity has a resiliency to lose 1 node and still be able to provide I/O through the virtual disks. A loss of a 2nd node will interrupt I/O which will automatically be restored nodes are available again. If your deployment needs to be resilient to 2 nodes being down (either failed or shutdown for maintenance), then you should configure the virtual disk to use only mirror. The first example below creates a mirror with parity volume and the 2nd example creates a mirror only volume.

Note

In the above command, the volume size must be specified with a size unit (such as GB or TB) and there must be no space between the numeric value and the size unit. For example, to create a 4TB virtual disk, you can specify it as 4TB or 4096GB.

The following PowerShell command creates a virtual disk with the mirror resiliency only:

#Create mirror virtual disks
New-Volume -StoragePoolFriendlyName <StoragePoolName> -FriendlyName <VirtualDiskName> -PhysicalDiskRedundancy 2 -FileSystem CSVFS_REFS –Size <Size in size units>

The New-Volume was introduced in Windows Server 2012 R2 and this command significantly simplifies deployments as it ties together a long list of operations that would otherwise have to be done in individual commands such as creating the virtual disk, partitioning and formatting the virtual disk, adding the virtual disk to the cluster, and converting it into CSVFS.

Step 8. Create a file server

Note

Skip this step for hyper-converged deployments.

After creating virtual disks, use the following PowerShell command to add the Scale-Out File Server role to the cluster. VMs located on the same cluster as Storage Spaces Direct (hyper-converged) should use the CSV namespace (c:\clusterstorage\volumeX) and not an network path..

New-StorageFileServer  -StorageSubSystemName <FQDN of the subsystem> -FriendlyName <ScaleOutFileServerRoleName> -HostName <ScaleOutFileServerRoleName> -Protocols SMB

One of the targeted use case in Windows Server 2016 Technical Preview is a Scale-Out File Server for storing Hyper-V virtual machines, whether those be primary virtual machines, replica virtual machines, or Data Protection Manager virtual machines for backup.

Note

The ScaleOutFileServerRoleName should be 15 characters or less.

Step 9. Create file shares

Note

Skip this step for hyper-converged deployments.

After adding the Scale-Out File Server role to the cluster, use the following PowerShell commands to create files shares on the virtual disks.

#Create file shares
md <path> 
New-SmbShare -Name <sharename> -Path <path> -FullAccess <Hyper-V Hosts>, <Hyper-V admins>, <Hyper-V Cluster>
Set-SmbPathAcl -ShareName <sharename>

Note

Hyper-V hosts are the computer name followed by the $ symbol, e.g. HyperV1$. Hyper-V admin is the user accounts for the users administering the Hyper-V servers that are using the file shares. Hyper-V Cluster is the Hyper-V cluster names followed by the $ symbol, e.g. VMCluster$. Also note that the file system ACL is automatically updated to the same as the share ACL.

Once you have created the file shares, you are now ready to deploy virtual machines files to the file shares.

Storage Spaces Optimize Pool

Windows Server 2016 Technical Preview Storage Spaces Direct can optimize a storage pool to balance data equally across the set of physical disks that comprise the pool.

Over time, as physical disks are added or removed or as data is written or deleted, the distribution of data among the set of physical disks that comprise the pool may become uneven. In some cases, this may result in certain physical disks becoming full while other disks in the same pool have much lower consumption.

Similarly, if new storage is added to the pool, optimizing the existing data to utilize the new storage will result in better storage efficiency across the pool and, potentially, improved performance from the newly available additional physical storage throughput. Optimizing the pool is a maintenance task which is performed by the administrator.

Limitations in Windows Server 2016 Technical Preview:

• Optimize pool is supported only with Simple or Mirror Spaces; Parity Spaces are not supported in Windows Server 2016 Technical Preview.

• There may be cases in Windows Server 2016 Technical Preview where the rebalance operation does not complete. If the rebalance job has stopped making progress, then it is likely that the job is stuck. You can check the status of the optimize job using the command described above. To resolve this condition, run the following command: Get-VirtualDisk | Repair-VirtualDisk.

You can optimize a storage pool with the following command:

Optimize-StoragePool <PoolName>

The output of the Optimize-StoragePool command include a progress bar that measures the progress of the re-balance operation

You can monitor the progress of the optimize job with the following command:

Get-StorageJob | ? Name –eq Optimize

Storage Spaces Fault Tolerance

Windows Server 2016 Technical Preview Storage Spaces Direct enhances the resiliency of virtual disks to enable resiliency to node failures. This is in addition to the existing disk and enclosure resiliency.

When using Storage Spaces Direct, storage pools and virtual disks will, by default, be resilient to node failures. When a storage pool is created the “FaultDomainAwarenessDefault” property is set to “StorageScaleUnit”. This controls the default for virtual disk creation. You can inspect the storage pool property by running the following command:

Get-StoragePool -FriendlyName <PoolName> | FL FriendlyName, Size, FaultDomainAwarenessDefault

FriendlyName                : <PoolName>
Size                        : <Size>
FaultDomainAwarenessDefault : StorageScaleUnit

Subsequently, when a virtual disk is created in a pool, it inherits the default value of the pool and therefore will have its “FaultDomainAwareness” set to “StorageScaleUnit”. You can inspect the virtual disk property by running the following command:

Get-VirtualDisk -FriendlyName <VirtualDiskName>| FL FriendlyName, Size, FaultDomainAwareness, ResiliencySettingName

FriendlyName          : <VirtualDiskName>
Size                  : <Size>
FaultDomainAwareness  : StorageScaleUnit
ResiliencySettingName : Mirror

Note

Do not change the FaultDomainAwarenessDefault and FaultDomainAwareness values in a Storage Spaces Direct deployment.

The FaultDomainAwareness property on the virtual disk controls data placement in all scenarios, including initial data allocation when creating the virtual disk, when repairing a virtual disk from a disk, enclosure or node failure, when rebalancing virtual disks in a storage pool. All of these operations will take into account the “StorageScaleUnit” fault domain and ensure that copies of data are placed on different nodes.

Let us examine some basics about virtual disks. A virtual disk consists of extents, each of which are 1GB in size. A 100GB virtual disk will therefore consist of 100 1GB extents. If the virtual disk is mirrored (using ResiliencySettingName) there will be multiple copies of each extent. The number of copies of the extent (using NumberOfDataCopes) can be two or three. All in all, a 100GB mirrored virtual disk with three data copes will consume 300 extents. The placement of extents is governed by the fault domain, which in Storage Spaces Direct is nodes (StorageScaleUnit), so the three copies of an extent (A) will be placed on three different storage nodes e.g. node 1, 2 and 3 in the diagram below. Another extent (B) of the same virtual disk might have its three copies placed on different nodes, e.g. 1, 3, and 4 and so on. This means that a virtual disk might have its extents distributed all storage nodes and the copies of each extent is placed on different nodes. Figure 6 below illustrates a four node deployment with a mirrored virtual disk with 3 copies and an example layout of extents:

FIGURE 6: Four nodes with a mirrored virtual disk with 3 copies space 

Understanding the above, let us review the various failure scenarios included in this section, and examine how Storage Spaces handles them.

• Scenario 1: One or more sectors on a disk has failed

• Scenario 2: A disk has failed

• Scenario 3: A disk is missing

• Scenario 4: Storage node restart or maintenance

• Scenario 5: Permanent storage node failure

Scenario 1: One or more sectors on a disk has failed

In this scenario, Storage Spaces will reallocate the extent that is affected by the failing sectors. The target for the reallocation could be another disk in the same node, or another disk in another node that does not already have a copy of the extent. So if the three copies of the extent are on node A, B and C, and the extent on node A is affected by a sector failure, the new copy can be generated on a different disk in node A or any disk in Node D. Disks in node B and C cannot be used as these two nodes already have a copy of the extent.

Scenario 2: A disk has failed

In this scenario Storage Spaces will retire the physical disk from the storage pool when it discovers the disk has failed. After the physical disk has been retired, each virtual disk will start its repair process. Since the physical disk has been retired, the virtual disks will generate a new copy of the extents that were on the retired physical disk. The new copies will follow the same logic as in scenario 1.

Scenario 3: A disk is missing

In this scenario Storage Spaces will do one of two things:

• If the storage node or the physical enclosure to which the physical disk is attached is also missing, then Storage Spaces will not retire the physical disk.

• If however only the physical disk is missing, then Storage Spaces will retire the disk.

  The reason for #1 above is that during a storage node restart or temporary maintenance of a storage node all the disks and physical enclosures associated with that node will be reported missing. Automatically retiring all those disks and enclosures would potentially result in a massive amount of repair activity as all extents on those disks would have to be rebuild elsewhere in the storage system – this could easily be multiple terabytes of data. If the disks and enclosures are really missing and will not come back to the storage system, the administrator will need to retire the missing physical disks and start the repair process.

Scenario 4: Storage node restart or maintenance

In this scenario Storage Spaces will not automatically retire physical disks from the storage pool for the reasons described above in scenario 3. Once the storage node comes back online, Storage Spaces will automatically update all extents that are not up to date with the copies that were not affected by the restart or maintenance.

Scenario 5: Permanent storage node failure

In this scenario, Storage Spaces will require the administrator to retire all the affected physical disks from the storage pool, add additional storage nodes to the storage system if needed, and then start repair. The reason this not an automatic process is that Storage Spaces does not know if it is a temporary or permanent failure. It is not desirable to initiate a repair that could potentially result in significant I/O and CPU activity.

Storage Spaces Direct Monitoring

Windows Server 2016 Technical Preview includes new cmdlets which surface health, performance, and capacity information from the Storage Health Service to facilitate an abstracted monitoring experience that is user-friendly and sufficient for many basic scenarios. Faults reflect health problems and their severity and can be queried using the following PowerShell cmdlets:

• Debug-StorageSubsystem This PowerShell cmdlet shows any faults that affect the whole storage subsystem. Faults from nodes, enclosures, and physical disks are displayed here. For the most part, such faults are physical in nature and require user intervention, e.g. to replace a failed physical disk. You can use the cmdlet as follows:

  get-storagesubsystem clus* | debug-storagesubsystem
  

  Each fault also includes recommended next steps, which can be viewed as follows:

  get-storagesubsystem clus* | debug-storagesubsystem | select RecommendedActions
  

• Debug-FileShare and Debug-Volume This cmdlet shows faults that impact only specific consumption points. These are mostly logical in nature (such as virtual disk data degradation or quality of service issues). Many logical faults will resolve automatically, especially after the corresponding physical faults have been resolved. You can use the cmdlet as follows:

  get-fileshare -name <share name>  | debug-fileshare
  
  get-volume -filesystemlabel <Volume label> | debug-volume
  

• Collecting aggregate performance and capacity metrics is now simple. and can be accomplished using PowerShell.

  Get-StorageHealthReport This cmdlet returns reports with aggregate performance metrics such as IO/s, throughput, and latency as well as total and remaining capacity for the specified entity. Reports are currently available for the whole storage subsystem, individual nodes, and volumes. Today, the output displays the raw values; formatting improvements are forthcoming in future releases. You can use the cmdlet as follows:

  get-storagesubsystem clus* | get-storagehealthreport -Count <repetiton count>
  
  get-volume -filesystemlabel <Volume label> | get-storagehealthreport -Count <repetiton count>
  
  get-storagenode -Name "<Node FQDN>" | get-storagehealthreport -Count <repetiton count>
  

• The Storage Spaces Fault Tolerance section of this guide alludes to considerable new automation. For example, failed or lost physical disks are autonomously retired and removed from their pools. Replacement disks are autonomously added in their place. The requisite repair and rebuild jobs are autonomously started. This automation can be monitored as follows:

  Get-StorageHealthAction This cmdlet shows relevant scheduled, in-progress, or recently completed jobs. Today, the strings and presentation are rough; improvements are forthcoming in future releases. You can use the cmdlet as follows:

  get-storagehealthaction
  

Windows PowerShell example for all-flash configurations

This section includes steps to configure all-flash (NVMe SSD + SATA SSD) and multi-resilient virtual disks using PowerShell:

• Step 4. Enable Storage Spaces Direct

  # Enable Storage Spaces Direct for NVMe + SATA SSD 
  Enable-ClusterS2D –S2DCacheDevice NVMe
  

• Step 5. Create storage pools

  # Create storage pool
  New-StoragePool -StorageSubSystemFriendlyName <FQDN of the subsystem> -FriendlyName <StoragePoolName> -ProvisioningTypeDefault Fixed -PhysicalDisk (Get-PhysicalDisk | ? CanPool -eq $true)
  
  Get-StoragePool <StoragePoolName> | Get-PhysicalDisk |? BusType -eq NVMe | Set-PhysicalDisk -Usage Journal
  

• Step 6. Create storage tiers

  # Define Storage Tiers
  $mt = New-StorageTier -StoragePoolFriendlyName <StoragePoolName> -FriendlyName <MirrorTierName> -MediaType SSD -ResiliencySettingName Mirror -NumberOfColumns 4 -PhysicalDiskRedundancy 2
  $pt = New-StorageTier -StoragePoolFriendlyName <StoragePoolName> -FriendlyName <ParityTierName> -MediaType SSD -ResiliencySettingName Parity -NumberOfColumns 7 -PhysicalDiskRedundancy 2
  

  Note

  If you have more than 10 capacity devices per node, you can change the number of columns to 8 for the mirror tier.

• Step 7. Create virtual disks

  New-Volume -StoragePoolFriendlyName <StoragePoolName> -FriendlyName <VirtualDiskName> -FileSystem CSVFS_ReFS -StorageTiers $mt,$pt -StorageTierSizes 100GB,800GB
  

Appendix

Understanding the difference between “Storage Spaces with Shared JBODs” and “Storage Spaces Direct”

To help understand Storage Spaces Direct, let’s examine Storage Spaces in Windows Server 2012 R2 highly available storage systems. In Windows Server 2012 R2, an HA system using Storage Spaces requires disks to be physically connected to all storage nodes. To allow for the disks to be physically connected to all storage nodes, they need to reside in an external JBOD chassis with each storage node having physical connectivity to the external JBOD. Also, only SAS disk devices are supported because, unlike SATA, SAS supports multi-initiator (connection from multiple servers). Because of these requirements, we have nicknamed this deployment “Storage Spaces with Shared JBODs” in contrast with Storage Spaces Direct. Figure 7 illustrates a “Storage Spaces with Shared JBODs” deployment.

FIGURE 7: Storage Spaces with shared JBODs

“Storage Spaces with Shared JBODs” provides many benefits compared to past highly available storage systems. However, requiring that disk devices are physically connected to every single node limits the type of disk devices that can be used and can lead to complex SAS fabric configurations, especially as these deployments scale out.

See Also

• Enabling Private Cloud Storage Using Servers with Local Disks (video)

• Testing Storage Spaces Direct using Windows Server 2016 virtual machines (blog)

• Storage Replica in Windows Server 2016 Technical Preview

• Storage Quality of Service