Node and Nodearray Objects

Node and nodearray objects are rank 2, and subordinate to cluster. A node represents a single Virtual Machine, whereas a nodearray can represent a collection of Virtual Machines, or at least one Virtual Machine scale set.

Node Defaults

The [[node defaults]] is a special abstract node that specifies the default setting for all nodes and nodearrays in a cluster:

[cluster my-cluster]
  [[node defaults]]
  Credentials = $Credentials
  SubnetId = my-rg/my-vnet/my-subnet
  MachineType = Standard_D2s_v3

  [[nodearray grid]]
  ImageName = cycle.image.centos6
  MachineType = Standard_H16

The $Credentials is a reference to a parameter named "Credentials".

In my-cluster, the grid nodearray inherits the Credential and SubnetId from the node defaults, but uses a specific HPC VM size of Standard_H16.

Example

This example template creates a cluster with two nodes and a nodearray. The proxy node uses the IsReturnProxy to define the special role of ReturnProxy, which will be the endpoint for a reverse channel proxy coming from CycleCloud when the cluster starts.

[cluster my-cluster]

  [[node defaults]]
    Credentials = $Credentials
    SubnetId = $SubnetId
    KeyPairLocation = ~/.ssh/cyclecloud.pem
    ImageName = cycle.image.centos7

  [[node proxy]]
    IsReturnProxy = true
    MachineType = Standard_B2

  [[node scheduler]]
    MachineType = Standard_D4s_v3

  [[nodearray execute]]
    MachineType = Standard_D16s_v3

Required Attribute Reference

There are a minimum of four required attributes to successfully start a node:

Attribute Type Definition
MachineType String The Azure VM Size
SubnetId String Subnet definition in the form ${rg}/${vnet}/${subnet}
Credentials String Name of the Cloud Provider account.

The fourth required attribute is related to an image. An image attribute is required but there are several forms it can take - see Image Attributes.

Additional Attributes

Attribute Type Definition
ComputerName String Computer name for VM. If specified, overrides the system-generated name.
ComputerNamePrefix String Prefix pre-pended to system-generated computer names
Zone String (list) Availability Zone for VM or VMSS. Can be a list for VMSS. E.g. Zone = 1,3
KeyPairLocation Integer Where CycleCloud will find a SSH keypair on the local filesystem
KeepAlive Boolean If true, CycleCloud will prevent the termination of this node
Locker String Specify the name of the locker from which to download project specs. See Use Projects
Attribute Type Definition
ComputerName String Computer name for VM. If specified, overrides the system-generated name.
ComputerNamePrefix String Prefix pre-pended to system-generated computer names
EphemeralOSDisk Boolean Use ephemeral boot disk for VM, if supported
Zone String (list) Availability Zone for VM or VMSS. Can be a list for VMSS. E.g. Zone = 1,3
ProximityPlacementGroupId String The full id for the Proximity Placement Group to put this node in. Must start with /subscriptions/
PlacementGroupId String If set, this label is used to place this node in a single placement group with all other nodes that have a matching value for PlacementGroupId. This offers lower latency communication and is required to enable InfiniBand on VM sizes that support it. This is usually set by the scheduler as needed so it does not need to be manually specified.
KeyPairLocation Integer Where CycleCloud will find a SSH keypair on the local filesystem
KeepAlive Boolean If true, CycleCloud will prevent the termination of this node
Locker String Specify the name of the locker from which to download project specs. See Use Projects
BootDiagnosticsUri String Storage URI for boot diagnostics (example: https://mystorageaccount.blob.core.windows.net/), if specified. Storage charges will apply.
HybridBenefit Boolean If true, enables "Azure Hybrid Benefit" licensing for Windows VMs
EnableTerminateNotification (8.2.0+) Boolean If true, enables Terminate Notification to send events on VM deletion to the VM for local handling. This only applies to scaleset VMs.
TerminateNotificationTimeout (8.2.2+) Relative Time If terminate-notification is enabled, this controls how long VMs are given to handle the event before being deleted.
ThrottleCapacity (8.2.2+) Boolean If true, this nodearray will report 0 capacity to autoscalers for a default of 5 minutes after encountering a capacity issue
ThrottleCapacityTime (8.2.2+) Relative Time If ThrottleCapacity is enabled, this is how long to report 0 availability after capacity is constrained. Default is "5m".
HybridBenefitLicense (8.3.0+) String If HybridBenefit is true, this specifies the license to use: RHEL_BYOS, SLES_BYOS, or Windows_Server. Default is Windows_Server.
FlexScaleSetId (8.3.0+) String If set, this is the fully qualified id of a scaleset in Flex orchestration mode that is used for the VM for this node.
EncryptionAtHost (8.4.0+) Boolean If true, the virtual machine will have Encryption At Host enabled.
SecurityType (8.5.0+) String Sets the security type; either undefined, TrustedLaunch or ConfidentialVM
EnableSecureBoot (8.5.0+) Boolean Enables Secure Boot, if using Trusted Launch VMs or Confidential VMs.
EnableVTPM (8.5.0+) Boolean Enables Virtual Trusted Platform Module, if using Trusted Launch VMs or Confidential VMs.
ScaleSetUpgradePolicyMode (8.6.2+) String Specifies the scale set upgrade policy, which controls what happens to existing VMs in a scale set when the scale set template is modified outside CycleCloud. This should generally be enabled if using an automated tool to modify existing scale sets, such as Azure Policy. Note: this does not automatically upgrade the OS image. One of Automatic, Rolling, or Manual (the default).

Note

A Proximity Placement Group is a general Azure feature, and one must be created before it can be referenced on a node. This lets CycleCloud VMs be collocated with other Azure resources in that proximity placement group, but does not enable InfiniBand networking. In contrast, PlacementGroupId is an arbitrary string in CycleCloud used to group VMs for nodes into a single scaleset that is constrained to be under the same networking switch, but may not be collocated with other Azure resources. They can both be used together but this may reduce the number of VMs that can be allocated.

Image Attributes

The VM image is a required setting to launch a virtual machine. There are three valid forms of image definition: default CycleCloud image names, Marketplace image definitions and Image IDs.

ImageName

CycleCloud supports a number of default Marketplace images that are available for different OS flavors. These can be specified with an ImageName.

Attribute Type Definition
ImageName String Cycle-supported image name. cycle.image.[win2016, win2012, centos7, centos6, ubuntu16, ubuntu14]

Marketplace Images

Along with Cycle-managed Marketplace Images, any marketplace image can be used by specifying the Publisher, Offer, Sku and ImageVersion.

Attribute Type Definition
Azure.Publisher String Publisher of VM Marketplace image
Azure.Offer String Offer for VM Marketplace image
Azure.Sku String Sku of VM Marketplace image
Azure.ImageVersion String Image Version of Marketplace image.

Note

A Marketplace image can also be specified in the ImageName attribute, encoded as a URN in the form Publisher:Offer:Sku:ImageVersion.

Images With Custom Pricing Plan

Shared Image Gallery images that have a pricing plan attached require information about the plan to be used, unless that information is stored in the Shared Image Gallery image. That is specified with the ImagePlan attribute using the Publisher, Product, and Plan nested attributes.

Note

Using custom images with a pricing plan requires CycleCloud 8.0.2 or later.

ImageId

Alternatively, the resource ID of a VM image in the Credential's subscription can also be used:

Attribute Type Definition
ImageId String Resource ID of VM image

Image Attributes

Marketplace image and images defined by ImageIds need a few additional settings to properly configure the CycleCloud OS extension:

Attribute Type Definition
DownloadJetpack Boolean If false, CycleCloud will not download Jetpack from the storage account. Jetpack must already be installed. Note: only Linux nodes are supported. Defaults to true. Added in 8.4.1.
InstallJetpack Boolean If false, CycleCloud will not install Jetpack on new VMs. Defaults to true.
AwaitInstallation Boolean If false, CycleCloud will not wait for Jetpack to report installation details when the VM is created. Defaults to true.
JetpackPlatform String Jetpack installer platform to use: centos-7, centos-6, ubuntu-14.04, ubuntu-16.04, windows. Deprecated in 7.7.0.

Warning

Setting InstallJetpack or AwaitInstallation is not recommended. In addition, setting DownloadJetpack requires a custom image with correct version of Jetpack install and is only recommended for environments that are experiencing issues downloading from storage accounts.

Note

ImageId is used by default if multiple image definitions are included in a single node definition.

Alternative Image Sample

Here is a sample template using the three alternate image constructs for the nodes:

[cluster image-example]
  [[node defaults]]
    Credentials = $Credentials
    MachineType = Standard_D2_v3
    SubnetId = my-rg/my-vnet/my-subnet

  [[node cycle-image]]
    ImageName = cycle.image.ubuntu16

  [[node my-custom-vm-image]]
    ImageId = /subscriptions/9B16BFF1-879F-4DB3-A55E-8F8AC1E6D461/resourceGroups/my-rg/providers/Microsoft.Compute/images/jetpack-rhel7-1b1e3e93

    # Jetpack already installed on image
    DownloadJetpack = false

  [[node marketplace-vm-image]]
    Azure.Publisher = Canonical
    Azure.Offer = UbuntuServer
    Azure.Sku = 16.04-LTS
    Azure.ImageVersion = latest

  [[node custom-marketplace-vm-image]]
    ImageName = /subscriptions/9B16BFF1-879F-4DB3-A55E-8F8AC1E6D461/resourceGroups/my-rg/providers/Microsoft.Compute/images/jetpack-rhel8-1b1e3e93
    ImagePlan.Name = rhel-lvm8
    ImagePlan.Publisher = redhat
    ImagePlan.Product = rhel-byos

Advanced Networking Attributes

Attribute Type Definition
IsReturnProxy Boolean Establish reverse channel proxy to this node. Only one node per cluster may have this setting as true.
ReturnPath.Hostname Hostname Hostname where node can reach CycleCloud.
ReturnPath.WebserverPort Integer Webserver port where node can reach CycleCloud.
ReturnPath.BrokerPort Integer Broker where node can reach CycleCloud.

Tags

CycleCloud supports tagging VMs and VMSS.

Attribute String Definition
Tags String Use tags.my-tag = my-tag-value to add tags to the deployment in addition to the tags assigned by CycleCloud by default.

Regular/Spot Attributes

CycleCloud supports the use of Spot VMs via the following attributes. See Spot Virtual Machines for more detail.

Attribute String Definition
Interruptible Boolean If true, the VM will be a Spot VM to provide reduced pricing.
MaxPrice Float The maximum price to spend on the VM. (Default: -1)

Nodearray-Specific Attributes

All of the attributes for a node are valid for a nodearray, but a node array is an elastic resource so additional attributes are available. Nodearray is a driver for Azure VirtualMachine ScaleSets (VMSS) and can have many backing VMSSs.

Attribute String Definition
Azure.AllocationMethod String Set this to StandAlone to manage single VMs or leave undefined to use VMSS.
Azure.SingleScaleset Boolean Use a single VMSS for all nodes (Default: false).
Azure.SinglePlacementGroup Boolean Use the single placement group setting for the VMSS. (Default: false)
Azure.Overprovision Boolean Use the Overprovision feature of VMSS. Cyclecloud will dynamically set depending on the scenario. This is an override.
Azure.MaxScaleSetSize Integer Limit the number of VMs in a single VMSS. Once this maximum is reached, CycleCloud will add additional VMSS to cluster. (Default: `40`)
InitialCount Integer Number of nodes to start when cluster starts.
MaxCount Integer To ensure that the cluster never exceeds 10 nodes you would specify a value of 10. Note that MaxCount and MaxCoreCount can be used together, in which case the lower effective constraint will take effect.
InitialCoreCount Integer Number of cores to start when cluster starts.
MaxCoreCount Integer To ensure that the cluster never exceeds 100 cores you would specify a value of 100. Note that MaxCount and MaxCoreCount can be used together, in which case the lower effective constraint will take effect.
ShutdownPolicy String Indicates what to do with the VM when a node shuts down. If 'terminate' the VM is deleted when the node shuts down. If 'deallocate', the node is stopped instead. (Default: terminate)
Attribute String Definition
Azure.AllocationMethod String Set this to StandAlone to manage single VMs or leave undefined to use VMSS.
Azure.SingleScaleset Boolean Use a single VMSS for all nodes (Default: false).
Azure.SinglePlacementGroup Boolean Use the single placement group setting for the VMSS. (Default: false)
Azure.Overprovision Boolean Use the Overprovision feature of VMSS. Cyclecloud will dynamically set depending on the scenario. This is an override.
Azure.MaxScaleSetSize Integer Limit the number of VMs in a single VMSS. Once this maximum is reached, CycleCloud will add additional VMSS to cluster. (Default: `40`)
InitialCount Integer Number of nodes to start when cluster starts.
MaxCount Integer To ensure that the cluster never exceeds 10 nodes you would specify a value of 10. Note that MaxCount and MaxCoreCount can be used together, in which case the lower effective constraint will take effect.
InitialCoreCount Integer Number of cores to start when cluster starts.
MaxCoreCount Integer To ensure that the cluster never exceeds 100 cores you would specify a value of 100. Note that MaxCount and MaxCoreCount can be used together, in which case the lower effective constraint will take effect.
ShutdownPolicy String Indicates what to do with the VM when a node shuts down. If 'terminate' the VM is deleted when the node shuts down. If 'deallocate', the node is stopped instead. (Default: terminate)
ThrottleCapacity Boolean Whether to suspend requests to Azure upon receiving Insufficient Capacity signal. (Default: false)
ThrottleCapacityTime Relative Time Backoff time after receiving Insufficient Capacity signal from Azure. AvailableCount will be reported as zero during this time. (Default: `5m`)

Note

All VMSSs will be assigned FaultDomainCount = 1

Inheritance

Nodes and nodearrays which are closely related can be derived from other nodes in the same cluster template. These inherited definitions minimize the declarations needed by sharing common attributes. Commonly used is the [[node defaults]] section, which is a special abstract definition that applies to all nodes and nodearrays in the cluster.

Attribute String Definition
Abstract Boolean If true, don't create an node or nodearray in the cluster. The abstract can be used for inheritance. (Default: false)
Extends String (list) Ordered list of inherited node/nodearray names. Items later in the list take precedence when values conflict. 'defaults' node will always effectively be first in the list. (Default: [])

Subordinate Objects

The node/nodearray objects have volume, network-interface, cluster-init, input-endpoint, and configuration as subordinate objects.