Deploy NV6ads_A10_v5 Cant get GPU

Question

When I deploy the online endpoint by VM NV6ads_A10_v5, But the Record logs show that only can use CPU, I have no idea about this!

Answer 1

Hi TaiHaoMed

Thank you for clarifying that you are not able to detect GPU on NV6ads_A10_v5 compute and it is only detecting CPU.We might need to check GPU driver availability and re-install the GPU drivers if needed.

nvidia-smi #to get get GPU availability

Install GPU drivers for Linux or windows

Once GPU availability is verified, we can force the registered model to use GPU with below command.

For Torch models

# Assuming 'model' is your registered model
import torch
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)

For Tensorflow models

tf.debugging.set_log_device_placement(True)

try:
  # Specify an invalid GPU device
  with tf.device('/device:GPU:2'):
    a = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]])
    b = tf.constant([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]])
    c = tf.matmul(a, b)
except RuntimeError as e:
  print(e)

Reference thread

Hope it helps.

Thank you.