Edit

Share via

Jetpack

You need Jetpack on every node of a cluster. Azure CycleCloud automatically installs Jetpack on each virtual machine that you provision to become a node in a cluster. Jetpack provides three main functions:

  • Node Configuration - CycleCloud uses scripts and Chef to automate the configuration of a provisioned VM into a working cluster node. A Chef client and the necessary resources for the configuration of the VM are embedded within Jetpack.
  • Distributed Synchronization - Jetpack manages communication between the node and the CycleCloud application server. This management enables CycleCloud to monitor the status of the provisioning VMs and synchronize the orchestration of multiple nodes in the cluster.
  • HealthCheck - Jetpack uses HealthCheck to determine the health of VMs so that it can terminate unhealthy VMs.

Jetpack Installation

The Jetpack installer is cached in your Azure Storage Account when you first start a cluster using CycleCloud. As cluster VMs are provisioned, a custom script extension is executed as part of the boot process. This extension downloads the Jetpack installer from your Azure Storage cache and then installs it on the VM.

The Jetpack installer:

  • Unpacks the Jetpack files to a single directory tree:
    • Windows: C:\cycle\jetpack
    • Linux: /opt/cycle/Jetpack
  • Creates system init startup scripts that configure a VM as a cluster node
  • Installs the HealthCheck service
  • Installs the Jetpack Command Line Tool to:
    • Windows: C:\cycle\jetpack\bin\jetpack
    • Linux: /opt/cycle/jetpack/bin/jetpack
  • Creates udev rules on Linux
  • Sets the environment variable CYCLECLOUD_HOME

Note

If Jetpack is preinstalled on the image, the custom script extension doesn't reinstall Jetpack. Instead, it runs an initialization step that validates the node's connection to CycleCloud and starts the healthcheck and jetpackd services before continuing to configure the node.

Jetpack Subdirectories

Directory Description
bin Useful binaries and scripts.
config User-defined and cluster-defined configuration files and scripts.
logs Logs generated by joining a cluster and converging the node. Of particular interest is the chef-client.log, which contains the results from converging Chef recipes.
run Runtime files generated by the system. We don't recommend directly accessing these files.
system Internal files. We don't recommend directly using any files in this directory as they can significantly change from release to release.

HealthCheck

The HealthCheck service executes user-defined scripts to determine a VM's current viability as a cluster node. For more information, see the HealthCheck documentation.

Jetpack Command Line Tool

The Jetpack command-line tool provides a useful set of subcommands for manipulating the current VM and interacting with Azure CycleCloud.

Command Description
jetpack autoscale Autoscale the cluster this node belongs to.
jetpack config Retrieve a configuration value.
jetpack converge Execute a Chef converge.
jetpack download Download a blob resource from a project in Azure Storage.
jetpack keepalive Delay system termination by the HealthCheck Service.
jetpack log Log a message to CycleCloud cluster UI.
jetpack run_on_shutdown Add a script to call before node termination.
jetpack send Send an arbitrary AMQP message to the CycleCloud server.
jetpack shutdown Request the shutdown of the VM by CycleCloud.
jetpack test Run tests associated with projects assigned to the VM.
jetpack users List users that CycleCloud manages on this VM.
jetpack report_issue Archives log files from VM to Azure Storage

jetpack autoscale

Use jetpack autoscale to set the autoscaling targets for the cluster that the node belongs to. You can scale clusters by cores, instance count, or custom definitions.

To scale to 100 cores:

jetpack autoscale --corecount=100

To scale the 'gpu' nodearray to five nodes:

jetpack autoscale --instancecount 5 --name=gpu

To customize autoscale, write a JSON file to disk that contains the nodearray definition you want to scale. To scale by 100 cores:

[
  {
      "Name": "execute",
      "TargetCoreCount": 100
  }
]

jetpack autoscale --file=custom-autoscale.json

jetpack config

Use jetpack config to fetch information that CycleCloud passes into a VM. It exposes:

  • all the system properties made available via Ohai
  • a subset of the VM's Azure metadata
  • information about the parent CycleCloud cluster.

jetpack converge

jetpack converge command downloads all CycleCloud projects associated with the node and starts a Chef converge process that runs all the Chef recipes and cluster-init scripts for the node.

jetpack download

jetpack download command downloads a blob that you uploaded with a project to the node. You must specify the project the blob belongs to.

To download the blob big-file.zip that you uploaded as part of the example-project project to the current directory:

jetpack download --project example-project big-file.zip .

jetpack keepalive

jetpack keepalive command interacts with the HealthCheck service to delay the termination of the VM due to a failing HealthCheck. You can set the delay to a fixed period or indefinitely. By default, the command delays termination for one hour.

To delay system termination by one hour:

jetpack keepalive

To delay system termination by six hours:

jetpack keepalive 6h

To disable the HealthCheck service entirely, that is, to delay termination indefinitely:

jetpack keepalive forever

Note

Only the forever option is available for HealthCheck on Windows VMs.

jetpack log

jetpack log sends a log message back to CycleCloud. The message appears in the application server log (typically /opt/cycle_server/cycle_server.log), the main event log, and the Cluster UI page.

Each message has two properties: level and priority.

The level property indicates the type of message. Valid levels are info, warn, and error. The level doesn't indicate the importance of a given message. For example, some errors are trivial and some informational messages critical.

Priority indicates the importance of the message. Valid priority values are 'low', 'medium', and 'high'. Only messages with a priority of medium or higher are displayed on the Cluster UI page to avoid flooding the page with low priority messages.

To send an informational log message that appears on the Cluster UI page:

jetpack log 'system is now ready'

To send a low-priority log message that you don't want to appear on the Cluster UI page:

jetpack log 'system is now ready' --priority low

By default, messages with a level of error have a high priority. To send an error message:

jetpack log 'the machine cannot process jobs' --level error

To send a trivial error message:

jetpack log 'the machine cannot process jobs' --level error --priority low

jetpack run_on_shutdown

The jetpack run_on_shutdown command registers a bash script to call before node termination.

The command takes the absolute path to the script as an argument.

When Azure terminates the node and Termination Notifications are enabled, Jetpack receives notification of the termination. It attempts to run the script before the node shuts down.

Nodes must enable Termination Notifications to enable run_on_shutdown.

jetpack run_on_shutdown /tmp/example.sh

This command isn't supported for Windows nodes.

jetpack send

jetpack send sends an AMQP message to CycleCloud. It's an advanced command that we don't recommend unless you're developing plugins for CycleCloud.

You can send arbitrary strings or files with specified AMQP routing keys.

jetpack shutdown

Use jetpack shutdown to request that CycleCloud terminate the node. You can pass options to the command to specify the reason for the shutdown request (idle vs unhealthy) and how to terminate the node (terminate vs deallocate).

To shut down an unhealthy node, use the following command:

jetpack shutdown --unhealthy

To deallocate the node:

jetpack shutdown --deallocate

jetpack test

Use jetpack test to run any tests that are included with projects assigned to the node. The command prints the results to stdout.

jetpack users

jetpack users lists the users that CycleCloud manages on the node. This list can change over time as users are assigned and removed to the cluster.

To get a user-friendly printout of the users assigned to the node:

$ jetpack users

Username: test-user
Full Name: Test User
UID: 10201
Is Admin: True
Is Owner: True

To get script-friendly JSON output:

$ jetpack users --json

[
    {
        "fullName": "Test User",
        "isAdmin": true,
        "isOwner": true,
        "name": "test-user",
        "publicKeys": [
            "ssh-rsa public-key-goes-here\n"
        ],
        "uid": 10201
    }
]

jetpack report_issue

jetpack report_issue archives log directories from the VM, optionally uploading them to Azure Storage and creating a signed URL for external access. The command uploads logs to the Azure Storage account referenced by the node's Locker. When you sign an archive in Azure Storage, the resulting SAS token has read only access for 30 days.

Usage:

$ jetpack report_issue [LOG_PATH] [--upload/--no-upload] [--sign/--no-sign]

To archive, upload, and sign the default Jetpack logs ($JETPACK_HOME/logs):

$ jetpack report_issue
Logs can be found at: https://testaccount.blob.core.windows.net/cyclecloud/issues/TestCluster-execute-1-77777964-8b74-420d-ad44-094edf7695f2.zip?sv=2017-11-09&rsct=binary&sig=jBJUlYo10lRq0eW94I%2B6syzYVmgo1qcTFUc35D/q0Tg%3D&se=2020-12-04T15%3A15%3A00Z&spr=https&rscd=disposition%3Dfile%3B%20attachment&sp=r&sr=b
Signed URL will expire on: 2020-12-04T15:15:00Z

To archive, upload, but not sign a nondefault log directory:

$ jetpack report_issue /var/log/azure --no-sign
Logs can be found at: https://testaccount.blob.core.windows.net/cyclecloud/issues/TestCluster-execute-1-d67fe991-1dac-4644-9af7-50c835726f5e.zip

To archive logs on the local VM:

$ jetpack report_issue --no-upload
Logs can be found at: /tmp/tmp4nscw705/TestCluster-execute-1-4249e973-3d87-4b14-94ed-6856a5267972.zip