Upgrade to HPC Pack 2019 Update 2

This article describes the steps to upgrade your HPC Pack cluster to HPC Pack 2019 Update 2.

Important

Before upgrading, please check if HPC Pack 2019 or HPC Pack 2019 Update 1 is installed. Open HPC Cluster Manager and click Help->About, the server version shall be 6.0.7205, 6.0.7214, or 6.1.7531. If you are still running HPC Pack 2016 with versions (5.x), you may reinstall the cluster with HPC Pack 2019 Update 2 slipstream.

Important

Downgrading from HPC Pack 2019 Update 2 to a previous version is not supported. If you upgrade your cluster to HPC Pack 2019 Update 2, you are not able to downgrade it back to the previous version after upgrading.

Preparation

Before the upgrading, you need to do the following:

Note

When upgrading a built-in high availability head node cluster that requires business continuity, step 1 - 4 are optional.

1. Stop all running jobs.

2. Stop all Azure PaaS nodes if you have deployed them.

3. Take all other nodes to offline.

4. Run the following command to determine what the current setting of the Azure Auto grow and shrink service.

   Add-PSSnapin Microsoft.Hpc
   Get-HpcClusterProperty -AutogrowShrink
   

   

   In this example the EnableGrowShrink is set to True or 1.

5. Run the following PowerShell command to stop the Azure auto grow and shrink service if it is enabled. Setting the value to 0, changes the value to False for the EnableGrowShrink property.

   Set-HpcClusterProperty -EnableGrowShrink 0
   

6. Optionally back up the HPC databases manually.

7. Download HPCPack2019Update2-Patch-v7756.zip from Microsoft Download Center. Unzip the package to your local folder of the head node(s). This zip file folder will be used to upgrade the head nodes and contains updates for the InstallShare folder described in the upcoming steps.

Upgrade the head node(s)

Important

For HPC Pack cluster with single head node, refer to Upgrade single head node. For HPC Pack cluster with multiple high availability head nodes, refer to Upgrade high availability head nodes

Upgrade single head node

To upgrade the head node for an HPC Pack cluster with single head node

1. Upgrade HPC components

Run HPCPack2019Update2_x64.exe as administrator on the head node, and reboot if required.

2. Replace the installation package in the install share

Unzip the InstallShare.zip file, and copy all the sub-folders and files to the folder %CCP_DATA%InstallShare and replace the files in the destination.

3. Continue to the next step to upgrade the compute nodes below.

Upgrade built-in high availability head nodes

This section is intended for upgrading the head node for an HPC Pack cluster with built-in high availability head nodes.

1. Upgrade HPC components

It is recommended to run the following PowerShell cmdlet to find the primary head node and do the upgrade on secondary head nodes first.

Get-HpcServicePrimary

Run HPCPack2019Update2_x64.exe as administrator on each of the head nodes, and reboot if required.

Click OK to continue.

 

Click Install.  

Click OK if prompted with the following dialogs.

Click Install.

2. Replace the installation package in the install share.

Locate the InstallShare.zip file that was included in the folder when downloading HPCPack2019Update2-Patch-v7756.zip.

Unzip the InstallShare.zip file, and copy all the sub-folders and files to the folder %CCP_DATA%InstallShare and replace the files in the destination.

Note

The InstallShare will only exist on initial primary head node and does not need to be copied to all head nodes.

3. Continue to updating the compute nodes section.

Upgrade Service Fabric high availability head nodes

To upgrade the head nodes for an HPC Pack cluster with Service Fabric high availability, do the following steps:

1. Upgrade the HPC Service Fabric application

Open a PowerShell console as administrator on any one head node, and run the following PowerShell command.

.\Upgrade-HpcApplication.ps1

Note

If you find it is stuck at "PreUpgradeSafetyCheck" due to some service fail to cancel, you could try to manually kill the corresponding process on the right node.

2. Upgrade HPC components on all three head nodes

Run HPCPack2019Update2_x64.exe as administrator on all three head nodes one by one.

3. Replace the installation package in the install share

Run the following PowerShell command as administrator to get the install share.

Add-PSSnapin Microsoft.Hpc
Get-HpcClusterRegistry | ?{$_.Name -eq "InstallShare"}

Output is similar to the following:

Name                                         Value
----                                         ----
InstallShare                                 \\HPCHN01\REMINST

Unzip the InstallShare.zip file, and copy all the sub-folders and files to the install share (i.e. "\\HPCHN01\REMINST") folder and replace the files in the destination.

4. (Optional) Upgrade the Service Fabric runtime

It is recommended to upgrade the Service Fabric runtime to the latest version. Run the following PowerShell commands on any one head node.

Connect to the cluster and get the list of available versions that you can upgrade to.

Connect-ServiceFabricCluster
Get-ServiceFabricRegisteredClusterCodeVersion

Start a cluster upgrade to the latest version from the list (for example 6.5.639.9590).

Start-ServiceFabricClusterUpgrade -Code -CodePackageVersion 6.5.639.9590 -Monitored -FailureAction Rollback

During the upgrading the original PowerShell console will close. Open a new one as administrator, connect to the Service Fabric cluster again with the Connect-ServiceFabricCluster command, and run the following command to monitor the upgrading progress.

Get-ServiceFabricClusterUpgrade

The upgrading completes when the UpgradeState becomes RollingForwardCompleted.

Upgrade Windows compute nodes, broker nodes, un-managed server nodes and workstation nodes

1. Copy HPCPack2019Update2_x64.exe (or HPCPack2019Update2_x86.exe for workstation node with 32-bit operating system) to a network share folder (such as \\HPCHN01\HPCUpdates) which everyone has read permission.
2. Open an command prompt console as administrator on one head node, and use the clusrun utility to upgrade the nodes. If some nodes roles are not part of the solution, this command may be skipped. Below is an example, update the share location as needed.

   clusrun /nodegroup:ComputeNodes \\HPCHN01\HPCUpdates\HPCPack2019Update2_x64.exe -unattend -SystemReboot
   
   clusrun /nodegroup:WCFBrokerNodes \\HPCHN01\HPCUpdates\HPCPack2019Update2_x64.exe -unattend -SystemReboot
   
   clusrun /nodegroup:WorkstationNodes \\HPCHN01\HPCUpdates\HPCPack2019Update2_x64.exe -unattend -SystemReboot
   
   clusrun /nodegroup:UnmanagedServerNodes \\HPCHN01\HPCUpdates\HPCPack2019Update2_x64.exe -unattend -SystemReboot
   

Note

If the cluster administrator doesn't have administrative privileges on workstation nodes and unmanaged server node, the clusrun utility may not be able to apply the update. In these cases the update should be performed by the administrator of the workstation and unmanaged servers.

Update Windows Client Machines

Client machines on-premises may not have access to Azure head node file share via SMB. In this case the installation zip file needs to be downloaded locally to install on any on-premises client machine utilizing HPC Pack Client Utilities.

Download the Hpc Pack Update 2 zip file locally and extract it to a local share that on-premises users have access to.

Run the HPCPack2019Update2_x64.exe or HPCPack2019Update2_x86.exe install to upgrade the Hpc Pack client utilities. This installation will upgrade the client utilities to Update 2. A reboot may be required.

The Installed Apps section of the client machine will show the HPC Pack 2019 Client Components version after a successful update as version 6.2.7756.0.

Upgrade Linux compute nodes

Upgrade on-premises Linux compute nodes

If your Linux compute nodes were installed manually as per Add Linux nodes to the cluster, use the following steps to upgrade.

Open HPC Cluster Manager on the head node, and click Resource Management > Nodes. Select all the Linux nodes, click Run Command, and run the following commands in sequence.

1. create a temp directory on all Linux nodes.

mkdir /tmp/hpc2019u2

2. mount the HPC installation share.

mount -t cifs //HPCHN01/REMINST/LinuxNodeAgent /tmp/hpc2019u2 -o vers=2.1,domain=<domainname>,username=<username>,password='<password>',dir_mode=0755,file_mode=0755

3. schedule a job on all Linux nodes to upgrade one minute later

cd /tmp/hpc2019u2; echo "python /tmp/hpc2019u2/setup.py -update" | at now + 1 minute

Upgrade Azure Linux compute nodes

If you had deployed the HPC Pack 2019 or HPC Pack 2019 Update 1 cluster with Linux workloads with our Azure Resource Manager template or Burst to Azure IaaS nodes feature, please follow the steps below to update the Azure Linux compute nodes,

1. Install Az PowerShell on Windows PowerShell on a head node.
2. Open a Windows PowerShell console as administrator on the head node, run Login-AzAccount to Log into Azure and then run the following PowerShell script,

$nodes = Get-HpcNode -GroupName LinuxNodes
foreach($node in $nodes)
{
    if([Version]::new($node.Version) -lt [Version]::new('16.2') -and $node.AzureInstanceName) {
        $linuxExt = Get-AzVMExtension -ResourceGroupName $node.AzureServiceName -VMName $node.AzureInstanceName | ?{$_.Publisher -eq 'Microsoft.HpcPack' -and $_.ExtensionType -eq 'LinuxNodeAgent2016U1'}
        if($null -ne $linuxExt)
        {
            Set-AzVMExtension -Publisher Microsoft.HpcPack -ExtensionType LinuxNodeAgent2016U1 -TypeHandlerVersion 16.2  -SettingString $linuxExt.PublicSettings -ResourceGroupName $linuxExt.ResourceGroupName -VMName $linuxExt.VMName -Name $linuxExt.Name -Location $linuxExt.Location -ForceRerun 1 -AsJob
        }
    }
} 

Upgrade Azure IaaS Windows compute nodes

Azure IaaS Windows compute nodes are by default deployed with “Microsoft ® HPC Pack 2019 ComputeNode Components" (You may go to Control Panel->Programs on the compute node to check it). For these Azure IaaS Windows compute nodes, please follow the steps below to apply the patch,

1. Download and copy “HpcCompute_x64.msi” and "UpdateCompactCN.ps1" to the remote install share on the head node \\<headnode>\REMINST or any file share that can be access by the compute nodes.
2. Clusrun /nodegroup:AzureIaaSNodes PowerShell.exe -ExecutionPolicy ByPass -Command "\\<headnode>\REMINST\UpdateCompactCN.ps1 -NewPackage \\<headnode>\REMINST\HpcCompute_x64.msi -RunAsScheduledTask"

Post Installation Notes

Reset PowerShell properties for EnableGrowShrink to what they were previously. In this example setting the value to 1 is setting this value to True.

Add-PSSnapin Microsoft.Hpc
Set-HpcClusterProperty -EnableGrowShrink 1

On the head node check the HPC Cluster Manager software version.