Migrate a Slurm cluster to a newer template

Azure CycleCloud application upgrades and Slurm cluster upgrades are separate operations. Upgrading the CycleCloud application doesn't replace the Slurm project, template, or software on cluster nodes. To adopt a newer Slurm project or template, create a replacement cluster, validate it, and then move workloads from the original cluster.

Important

CycleCloud doesn't provide an in-place Slurm template upgrade or template rollback operation. Keep the original cluster stopped or otherwise available until you validate the replacement cluster and complete the cutover.

Check version compatibility

Check the release notes for each component before you migrate. Their version numbers aren't interchangeable.

Component	Current release	What to verify
Azure CycleCloud	8.9.2	Review resolved issues and test the CycleCloud application upgrade in a nonproduction environment.
CycleCloud Workspace for Slurm	2026.08.07	This Workspace release supports CycleCloud 8.9.2. Use Workspace release notes only for Workspace deployments.
CycleCloud Slurm project and template	Azure/cyclecloud-slurm releases	Select a Slurm project release that supports your CycleCloud version, operating system, and required Slurm features.

For a standalone CycleCloud deployment, the Workspace version doesn't determine the Slurm project version. For a Workspace deployment, use a Workspace release that explicitly supports your CycleCloud version.

Prerequisites

Before you start:

Upgrade CycleCloud to the version required by the target Slurm project. Complete the application backup and staging validation described in that article.
Install and initialize the CycleCloud CLI.
Record integrations that aren't stored in cluster parameters, such as DNS aliases, monitoring, identity assignments, and external storage configuration.
Put custom templates, projects, cluster-init scripts, and configuration files in source control.
Plan a maintenance window to drain jobs and redirect submissions.
Confirm regional and VM-family quota for the replacement cluster. Quota doesn't reserve capacity.

Export the existing cluster parameters

Export the parameters that CycleCloud used to create the original cluster:

cyclecloud export_parameters <existing-cluster-name> -o existing-cluster-parameters.json

Store the exported file securely. It can contain environment-specific identifiers and settings. Review the file before you reuse it because the target template might add, remove, or rename parameters.

Choose a migration path

Use the stock-template path if the cluster uses a built-in Slurm template without template or project changes. Use the custom-template path if you changed the template, cluster-init project, scripts, or software configuration.

Migrate a stock Slurm template

Download the target Slurm project release from the CycleCloud Slurm repository. Review its release notes and template parameters.
From the downloaded project directory, upload the target project version to the configured locker:
```
cyclecloud project upload <locker-name>
```
Import the target template under a new name. Don't overwrite the template that the original cluster uses.
```
cyclecloud import_template <new-template-name> -f templates/slurm.txt
```
Compare existing-cluster-parameters.json with the target template parameters. Add required values and remove parameters that the target template doesn't support.

Create a replacement cluster with a new cluster name:

cyclecloud import_cluster <new-cluster-name> -c <template-name-in-file> -f templates/slurm.txt -p existing-cluster-parameters.json

Start and validate the replacement cluster before you change production job submission.

Migrate a customized Slurm template

Don't apply an old custom template directly over a newer stock template. Instead, rebase your customizations deliberately:

In source control, compare your current custom template and project with the stock files for both the current and target releases.
Copy the target stock template and project to a new branch or directory.
Reapply each required customization. Pay particular attention to:
- Template parameters, node arrays, VM images, and storage mounts.
- [[[cluster-init]]] references and their project, spec, and version values.
- Custom cluster-init scripts, Slurm configuration, and health checks.
- Managed identities, networking, monitoring, and scheduler integrations.
Set a new version=x.y.z value in the project's project.ini file. Project versions let the original and migrated cluster reference different project content.
Upload the new project version to the target locker.
```
cyclecloud project upload <locker-name>
```

Import the customized target template under a new name.

cyclecloud import_template <new-template-name> -f templates/<custom-template-file>

Review the exported parameters against the customized target template, and then create a replacement cluster with a new name.

cyclecloud import_cluster <new-cluster-name> -c <template-name-in-file> -f templates/<custom-template-file> -p existing-cluster-parameters.json

For more information about project lockers and versioned cluster-init references, see CycleCloud projects. For template syntax and parameter files, see Cluster templates.

Validate the replacement cluster

Validate the replacement cluster in a nonproduction environment before cutover:

Start the scheduler and a small execute node array.
Verify that scheduler, sign-in, and execute nodes complete cluster-init without errors.
Run sinfo and scontrol show nodes on the scheduler to confirm partitions, node states, features, and resources.
Submit representative serial and multi-node jobs. Confirm job output, shared storage access, and accounting behavior.
Scale each node array from zero and back to zero. Confirm that autoscaling, node health checks, and termination work as expected.
Verify monitoring, DNS, identity, proxy, and external service integrations.
Compare performance and application results with the original cluster.

If you use current H-series VMs for tightly coupled workloads, you can validate with Standard_HB368rs_v5 where it's available. This size has 368 vCPUs. Confirm image support, regional quota, VM-family quota, and capacity before you use the size for production.

Cut over and roll back

To cut over:

Stop new job submissions to the original cluster.
Drain running jobs and confirm that the queue is empty.
Synchronize any application data or configuration that changed during validation.
Redirect users, automation, and DNS aliases to the replacement cluster.
Monitor representative production jobs before you remove the original cluster.

To roll back before you delete the original cluster, stop submissions to the replacement cluster and redirect them to the original cluster. A CycleCloud server backup and restore protects the CycleCloud application and database, but it isn't a rollback mechanism for an individual cluster template or its nodes.

After the rollback window expires, terminate and delete the original cluster according to your organization's retention policy.

Troubleshoot migration issues

If the target template rejects the parameter file, compare its parameter names and types with the exported file. Don't use --force to replace the original cluster.
If cluster-init fails, confirm that each referenced project, spec, and version exists in the configured locker.
If execute nodes don't join Slurm, review the scheduler and node logs, and confirm that DNS, ports, and shared storage work from the replacement network.
If a current Slurm release changes configuration behavior, review the CycleCloud Slurm release notes and test the affected customization separately.

Feedback

Was this page helpful?

Last updated on 2026-09-11