Edit

Plan and size HPC clusters in Azure CycleCloud

Before you create a cluster in Azure CycleCloud, plan how the cluster is structured and sized. Good planning decisions up front - about the scheduler, the virtual machine (VM) types, how the cluster scales, and how it stores and moves data - determine how well the cluster performs and how much it costs to run.

This article describes the main decisions to make when you plan an HPC cluster in CycleCloud. It complements CycleCloud clusters and nodes, which explains what clusters, nodes, and node arrays are, and Create a new cluster, which walks through building a cluster from a template.

Choose a scheduler

CycleCloud is scheduler-agnostic. It has built-in support for the schedulers that most HPC teams already use, and it adds autoscaling to each of them:

  • Slurm
  • PBS Professional
  • IBM Spectrum LSF
  • Altair Grid Engine
  • HTCondor

Choose the scheduler that matches your existing workflows and job scripts so that you can move workloads to Azure with minimal change. If your organization uses a scheduler that isn't in the built-in list, you can integrate it with the CycleCloud autoscaling API. For more information, see Scheduling in CycleCloud.

Choose VM families and sizes

The VM family and size you select for compute nodes have the largest effect on both performance and cost. Match the VM to the characteristics of your workload:

  • Tightly coupled (MPI) workloads that exchange data between nodes benefit from HPC-optimized VMs with a low-latency InfiniBand interconnect, such as the HPC VM sizes.
  • GPU-accelerated workloads, such as AI training or rendering, use GPU-optimized VM sizes.
  • Embarrassingly parallel or throughput workloads, where tasks run independently, can use general-purpose or compute-optimized VMs and don't require a specialized interconnect.

For a full comparison of VM options, see Sizes for virtual machines in Azure.

Because a node array can span more than one virtual machine scale set, a single array can offer VMs of different sizes or families for the same role. This flexibility helps when your preferred VM size has limited capacity in a region. For more information, see CycleCloud clusters and nodes.

Size the cluster and plan for autoscaling

CycleCloud automatically scales the number of compute nodes based on the work queued in the scheduler, so you don't need to guess a fixed cluster size. Instead, plan the following boundaries:

  • Maximum size. The cluster form limits how far a cluster autoscales, based on the total number of cores it can start. Set this limit to reflect the largest workload you expect and the quota available in the region.
  • Dedicated versus Spot nodes. Dedicated nodes are reserved for your pool and give you predictable capacity. Spot nodes use surplus Azure capacity at a reduced price, but Azure can reclaim them when it needs the capacity back. Use dedicated nodes for time-sensitive or long-running jobs, and Spot nodes for fault-tolerant or restartable work where cost matters more than guaranteed availability.
  • Scale-down behavior. Autoscaling removes idle nodes so that you stop paying for compute you aren't using. Confirm that your jobs write results to durable storage, because local data on a node is lost when the node is deallocated.

Check quota, capacity, and region

The VMs you plan to use must be available, and within quota, in the region where you deploy:

  • Region. Choose a region that offers the VM families your workload needs and that's close to your data and users. Not every VM family is available in every region.
  • Quota. Core (vCPU) quotas are set per region and per VM family. Make sure your quota is high enough for the maximum cluster size you planned, and request a quota increase if needed.
  • Capacity. Even within quota, HPC VM sizes can be constrained in a region at a given time. Consider a fallback VM size or an additional region for large or time-critical runs.

Plan storage and networking

HPC jobs are often bound by how quickly they can read and write data, not just by compute:

  • Shared file systems. Most HPC clusters need a shared file system that all nodes can mount. CycleCloud can deploy NFS servers and parallel file systems (for example, BeeGFS) as part of the cluster, and you can connect to managed services such as Azure NetApp Files or Azure Managed Lustre.
  • Interconnect. Tightly coupled MPI jobs require the InfiniBand interconnect available on HPC-optimized VMs. Confirm that both the VM size and the region support it.
  • Networking. Plan the virtual network and subnets that host the cluster, and ensure the CycleCloud application server has the outbound access it needs to manage nodes. For production guidance, see Plan your production deployment.

Plan for cost

Compute costs are usually the largest expense in an HPC environment. Plan to control these costs by:

  • Using autoscaling and Spot nodes to avoid paying for idle capacity.
  • Right-sizing VMs to match the workload instead of defaulting to the largest option.
  • Tracking spending through the built-in integration with Microsoft Cost Management. For more information, see Cost and usage tracking.

Next steps

  • Last updated on