CCP_RESTART on HPC with Linux workstations

Question

CCP_RESTART on HPC with Linux workstations

Sven 21

Hi,

CCP_RESTART usage with Windows based machines on HPC cluster just seem to run fine.
Now having the need to include Linux machines that specific setting doesn't do the job.
Doing a trick around a "sudo reboot" command, just let's the machine reboot 3 times.
Afterwards the task is of state "Failed".

Now the question would be: will the reboot/restart feature be supported as well for Linux machines soon
or to get a workaround: how can I limit the automatic task requeue, setting "AutoRequeueCount" to e.g. 0 or 1
doesn't seem to do the job to let the machine just boot once only.

BR Sven

Answer accepted by question author

0 additional answers

Your answer

Answer 1

vipullag-MSFT 26,522 Moderator

@Sven

Apologies for the delayed response on this.

CCP_RESTART is supported only on Windows nodes. There is no recent update to support this on Linux nodes.

There are cluster configurations for job/task retry times (by default 3) for system failures.

Hope this helps.
Please 'Accept as answer' if the provided information is helpful, so that it can help others in the community looking for help on similar topics.

Sven 21 Reputation points

2021-10-15T06:55:53.81+00:00

Hi,
thanx for the answer and hint on it.
One more thing: can a job or task overwrite the Retry amount/number on job submission?
If so, how can that be (in xml) be configured?

BR Sven
vipullag-MSFT 26,522 Reputation points Moderator

2021-10-20T02:08:41.253+00:00

@Sven

The system failure auto retry count is configured at cluster level and cannot be overwritten at job level.

The task execution failure retry count can be set at the job level, as described below,

Job-level automatic task requeue after application execution failure
HPC Pack 2012 R2 Update 1 introduces an integer job property TaskExecutionFailureRetryLimit with default value 0. When set with a positive integer N, all the tasks in a job except node preparation and release tasks automatically requeue for a maximum of N times when an application execution failure happens (e.g., return code is not zero or not within the defined scope). This property is available in related job commands, HPC PowerShell cmdlets such as New-HpcJob, and the HPC Job Scheduler API. Previous versions of HPC Pack provide only cluster-wide automatic job or task requeue settings for a cluster scheduling or dispatching problem.
API example:
Scheduler s = new Scheduler();
s.Connect("HeadNode");
var job = s.CreateJob();
job.Name = "TestFailRetry";
job.TaskExecutionFailureRetryLimit = 3;
var task = job.CreateTask();
task.CommandLine = "ping localhost -n 5 && exit 1";
job.AddTask(task);
s.SubmitJob(job, "username", "password");

Share via

CCP_RESTART on HPC with Linux workstations

0 additional answers

Your answer