Share via

Selecting the number of nodes for your Scale-Out File Server Cluster

  • Article

I recently got a stream of e-mails and questions about the maximum number of cluster nodes you can have in a Scale-Out File Server cluster. For the record, we test and support up to 8 nodes per file server cluster. This is the case for both Windows Server 2012 (which introduced the Scale-Out File Server cluster feature) and Windows Server 2012 R2.

 

image

 

However, the real question is usually "How many file server nodes do I need for my Scale-Out File Server cluster?" The most common scenarios we see involve the deployment of 2 to 4 file server nodes per cluster, with just a few people considering 8 nodes. Here are some arguments for each cluster size:

  • 2 is the bare minimum for achieving continuous availability (transparent failover). We see a lot of these two-node clusters out there.
  • 3 is a good idea to allow you to still have continuous availability even when you're doing maintenance on a node.
  • 4 allows you to upgrade the cluster without extra hardware by evicting two nodes, installing the new operating system and using the Copy Cluster Roles Wizard.
  • 8 will allow you to combine the network throughput and computing power of the many nodes to create an amazing file sharing infrastructure.

On the topic of performance, keep in mind that the Scale-Out concept means that you have the ability to linearly scale the cluster by adding more nodes to achieve higher IOPS and throughput. We have proved it in our test labs. However, we have also shown that a single file server cluster node can deliver over 200,000 IOPs at 8KB each or over 2GB/sec throughput, with a fairly standard server configuration (two modern Intel CPUs with a few cores each, dual 10GbE RDMA network interfaces, two SAS port each with four lanes of 6Gbps SAS shared storage with an SSD tier). We have also shown high-end file server configurations achieving over a million IOPs and over 16.5GB/sec from a single node. So, there are only a few scenarios that would require anything close to 8 nodes solely to achieve specific performance goals.

Also, keep in mind that the reasons for adopting a Scale-Out File Server usually include a combination of availability, performance and scalability of the resulting file service. While the number of nodes is an important ingredient to achieve that, you should never overlook related configuration decisions like the number and model of CPU you're using, the number and type of network interfaces, the number and speed of the SAS ports, the proper deployment of tiered storage with the right class of SSDs and even the generation and number of lanes of the PCIe slots you use. Two well sized and properly fitted file server nodes can easily beat eight nodes that were poorly put together.

Here a few additional references for those looking to dig deeper:

I'm not sure what's driving this wave of questions about the Scale-Out File Server cluster limits, but I hope this helps clarify the topic. Have you ever encountered a scenario where you would need more than 8 nodes? We would love to hear it...

Comments

  • Anonymous
    November 08, 2013
    Another important item to consider with SAS storage is whether the server supports PCI express 3.0you need PCIe 3.0 x8 to support the bandwidth a dual port SAS card like the LSI 9207-8e can generate
  • Anonymous
    November 08, 2013
    @MichaelPCI Gen3 x8 goes up to about ~6.7 GB/sec, which can keep up with 2 ports of 6Gbps/4-lane SAS (total of 4.4 GB/sec from both ports combined). If using PCIe Gen2 x8 or PCIe Gen3 x4, the bus becomes a bottleneck.See tables at the end of blogs.technet.com/.../hyper-v-over-smb-performance-considerations.aspx for further details on speeds and feeds...
  • Anonymous
    November 08, 2013
    Hey Jose,Great article!!From my point of view the amount of RAM is another important decision for each node since you can setup up to 80% of RAM exclusively for cache on CSV volume.A question for you:  I guess is strongly recommended to have all nodes with the same hardware configuration to avoid unbalanced configuration, and surprises during fail-over process, am I right?
  • Anonymous
    November 08, 2013
    @AndreYes, RAM for CSV caching is important and balanced nodes is always a good idea to make things more predictable.To be fair, whatever passes Failover Cluster Validation is supported, but I would keep them configured exactly the same, like monozygotic twins.
  • Anonymous
    November 09, 2013
    Dear Jose,I have a question out of the above scope please.You mentioned that always we need to choose the smaller number of the two tiers SSD+HDD for the NumberOfColumns.The question if I added more disks in the JBOD to expand the storage space in the future.The example we have 4 disks in the SSD tier and 8 disks in the HDD tier for a mirrored space.I will add 4 SSDs and 4 HDDs (Total= 8 disks in the SSD tier and 12 disks in the HDD tier).How this will reflect the NumberOfColumns?Before we expand the JBOD, the NumberOfColumns was set to 2 for the mirrored space (4 SSD/2).Do I need to run a new cmdlet below and modify the NumberOfColumns from 2 to 4?Get-StoragePool Pool1 | Set-ResiliencySetting -Name Mirror -NumberOfColumnsDefault 4I am setting Storage Tiering on a new HP server with 25 Bays mixed between HDD and SATA SSD.I noticed that the Storage Pool detect the media type for the SSD disks as SSD, but for the HDD is showing as UNKNOWN.Is it safe to SET the PhysicalDisk -MediaType = HDD in production and not for lab test?Is it recommended that the Server must detect the media type of the disks by default or we can set it in PowerShell if it showing Unknown?Thank you in advance.Regards.
  • Anonymous
    November 10, 2013
    Can I use Single Share for Scale-Out File Server with minimam Redirection of CSV Data/Metadata at Mirrored Storage Space ?In your article "Automatic SMB Scale-Out Rebalancing in Windows Server 2012 R2",at section "Hyper-V over SMB with Windows Server 2012 R2 and Mirrored Storage Spaces (asymmetric)"" In this new release, the SMB client gained the flexibility to connect to different Scale-Out File Server cluster nodes for each independent share that it needs to access"AND, at section "Key configuration requirements for asymmetric storage in Windows Server 2012 R2"" To be clear, if you have a single CSV in such asymmetric configuration in Windows Server 2012 R2 Scale-Out File Server cluster, only one node will be actively accessed by SMB clients"I want to use Mirrored Stoage Space, and 4 Hyper-V Cluster node, 10 Guest OS per node,How many Share do I need ?Regards,I like Windows Server 2012 R2 Storage SpaceYoshihiro Kawabata
  • Anonymous
    November 10, 2013
    @YoshihiroIf you use a two-node Scale-Out File Server with a single share backed by a mirrored storage space in Windows Server 2012 R2, all clients (Hyper-V hosts) running Windows Server 2012 R2 will be redirected to a single file server node, the owner node for that single CSV. The other file server node will be idle waiting for a potential failover.This is not about the number of Hyper-V nodes, but the number of file server nodes in the configuration. You mentioned you had 4 Hyper-V nodes. Assuming you have 2 file server cluster nodes, you should use at least 2 shares, each backed by a different CSV on a mirrored storage space. Each of the 4 Hyper-V nodes will connect to the file server node that owns the CSV for that share. If a single Hyper-V node is using both shares, that node will connect to both file server nodes and use the best path for each share.
  • Anonymous
    November 10, 2013
    @SharbelIt's safe to set the media type property if it's not automatically detected. Just make sure you  set it to the correct value (HDD or SSD) and don't place any data on it before doing it.You can't change the number of columns for an existing space (the number of columns is a property of the storage space, not the storage pool). You can add disks to the pool to get more capacity and you can increase the size of the space (using Resize-VirtualDisk), even if it's tiered (use Resize-StorageTier). However, the number of columns has to remain the same, since it dictates how data is laid out on the disk.You can find the Storage Spaces FAQ at http://social.technet.microsoft.com/wiki/contents/articles/11382.storage-spaces-frequently-asked-questions-faq.aspx
  • Anonymous
    November 10, 2013
    What a brilliant post, much appreciated Jose a tonne of information there :)
  • Anonymous
    November 11, 2013
    @Jose Barreto -- Thank you for clear response.I will continue to try Windows Server 2012 R2 Scale Out File Server.Regards,
  • Anonymous
    November 11, 2013
    What you're doing with  SMB is refreshing. I look forward to seeing the move towards leveraging the physical desktop and I hope youre involved.
  • Anonymous
    November 11, 2013
    Thank you dear Jose for the clear response,I am having an issue to (Resize-StorageTier) and expand the SSD_Tier:I still have 7GB free on the SSD Tier:PS C:Get-StorageTierSupportedSize MirroredTieredSpace1_SSD_Tier -ResiliencySettingName Mirror | FT -AutoSizeSupportedSizes TierSizeMin TierSizeMax TierSizeDivisor--------------        -----------   ----------- ---------------{}                    1073741824  7516192768      1073741824When I try to expand the SSD tier using below cmdlet:PS C:Resize-StorageTier -InputObject (Get-StorageTier -FriendlyName "MirroredTieredSpace1_SSD_Tier") -Size 3GBResize-StorageTier : Invalid ParameterAt line:1 char:1+ Resize-StorageTier -InputObject (Get-StorageTier -FriendlyName "MirroredTieredSp ...+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~   + CategoryInfo          : InvalidArgument: (StorageWMI:ROOT/Microsoft/...SFT_StorageTier) [Resize-StorageTier], CimException   + FullyQualifiedErrorId : StorageWMI 5,Resize-StorageTierPS C:Get-StorageTier Space1ObjectId             : {1}\ISCKC-HV01root/Microsoft/Windows/Storage/Providers_v2SPACES_StorageTier.ObjectId="{6fa45e22-498a-11e3-80b4-806e6f6e6963}:ST:{5c3a3723-4ab2-11e3-80bb-38eaa78c3d44}{5c3a37ae-4ab2-11e3-80bb-38eaa78c3d44}"PassThroughClass     :PassThroughIds       :PassThroughNamespace :PassThroughServer    :UniqueId             : {5c3a37ae-4ab2-11e3-80bb-38eaa78c3d44}Description          :FriendlyName         : MirroredTieredSpace1_SSD_TierMediaType            : SSDSize                 : 375809638400PSComputerName       :ObjectId             : {1}\ISCKC-HV01root/Microsoft/Windows/Storage/Providers_v2SPACES_StorageTier.ObjectId="{6fa45e22-498a-11e3-80b4-806e6f6e6963}:ST:{5c3a3723-4ab2-11e3-80bb-38eaa78c3d44}{5c3a37af-4ab2-11e3-80bb-38eaa78c3d44}"PassThroughClass     :PassThroughIds       :PassThroughNamespace :PassThroughServer    :UniqueId             : {5c3a37af-4ab2-11e3-80bb-38eaa78c3d44}Description          :FriendlyName         : MirroredTieredSpace1_HDD_TierMediaType            : HDDSize                 : 2394444267520PSComputerName       :Thank you in advance.Regards,
  • Anonymous
    November 11, 2013
    @SharbelOn the Resize cmdlet, you have to specify the new total size, not the amount you want to increase.You can find an example of how to use Resize-StorageTier (and the required Update-Disk) in item 22 of this blog post:  blogs.technet.com/.../step-by-step-for-storage-spaces-tiering-in-windows-server-2012-r2.aspx
  • Anonymous
    November 13, 2013
    The comment has been removed
  • Anonymous
    November 13, 2013
    @GünterYou can have a mirror with disks on the same JBOD.To mirror across 2 different JBODs, they must be connected to the same hosts.If there are different hosts connected to different JBODs, you can use Hyper-V Replica.
  • Anonymous
    November 19, 2013
    Thank you dear Jose for your prompt reply.I have a Server with 24 bays (18 HDDs + 6 SSDs). What is the best practices to create a Storage Space with tiering (Mirror) to store Virtual Machines? One virtual disk as one volume for all the 24 disks or two virtual disks 2 separate (Mirror each volume) for better performance?Regards,
  • Anonymous
    November 19, 2013
    @SharbelYou can probably do a single pool and a few virtual disks.You generally should keep your virtual disks smaller than 64TB (there is a limit related to snapshots). If you're using large disks (4TB) and 2-way mirroring, you'll likely have 48TB of usable disk space, so a single virtual disk is a possibility.However, in a Windows Server 2012 R2 Scale-Out File Server cluster, you want to have at least as many virtual disks as you have nodes. So, for a two-node cluster and the same disk sizes, you could use two virtual disks using 24TB each.
  • Anonymous
    November 23, 2013
    Great Info! I'm building this Scale out File Server scenario now in our lab. A general question. I have three network cards per SoFS node: a dual RDMA (iWARP) card and two 1GbE cards. Should the RDMA nics have IPs on my CorpNet lan? Or should my RDMA nics each have dedicated networks?I'm guessing I also want a private heartbeat network as well. Pretty cool architecting this stuff.
  • Anonymous
    November 23, 2013
    The comment has been removed
  • Anonymous
    November 26, 2013
    Thanks Jose. Follow up question. So if both my RDMA nics are on my corporate network, then I have only a single network listed under Failover Cluster Networks? Would that be correct? So for my storage cluster, all of my nics would be on my corporate network (except iSCSI networks)?
  • Anonymous
    March 30, 2014
    In this post, I'm providing a reference to the most relevant content related to Windows Server 2012
  • Anonymous
    July 07, 2014
    In this post, I'm providing a reference to the most relevant content related to Windows Server 2012