How to fix the error "failed to create shim task" after submitting a job to compute cluster?

Lu 66

As I submitted a job, it failed and gave me the following messages:

Service Error:

Failed to execute command group with error Docker responded with status code 500: {"message":"failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'\nnvidia-container-cli: initialization error: nvml error: driver not loaded: unknown"}

Warning:

AzureMLCompute job failed. OrchestrateJobError: Failed to execute command group with error Docker responded with status code 500: {"message":"failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: err

YutongTie-MSFT 48,586 Reputation points

2023-06-22T21:47:37.3333333+00:00

Hello @Lu

Thanks for reaching out to us for this issue, the error message you received indicates that there was an issue with the Docker container that was created for your job. Specifically, it seems that the container failed to start due to an error with the NVIDIA driver.

To fix this issue, you can try the following steps**:**

Check that the NVIDIA driver is installed on the compute cluster. You can do this by logging into the compute cluster and running the command nvidia-smi. If the command fails or returns an error, then the NVIDIA driver is not installed correctly.

If the NVIDIA driver is not installed, you will need to install it on the compute cluster. You can follow the instructions provided by NVIDIA to install the driver on the specific operating system and hardware configuration of your compute cluster.

If you are still seeing this issue, please let us know.

Share via

How to fix the error "failed to create shim task" after submitting a job to compute cluster?