How can I set --ulimit memlock=-1
Daniel Otero Gómez
0
Reputation points
I tried specifying it in the docker args but it didn't let me build the container. What I am trying to do is troubleshoot the shared memory of the VM like suggested [here]
How can I do this then?
I know there is a separate argument for the shm-size within the command function and I am working with it, but still, I would like to troubleshoot the memory limit as specified in the documentation.
# Create or update the component
print("Creating job...")
print(job_command)
command_job = command(
experiment_name="testing-ssl-byol",
description=description,
code=str(code_dir),
environment=enviornment,
inputs=inputs,
outputs=outputs,
command=job_command,
compute="Testing-GPU-Cluster",
distribution=MpiDistribution(process_count_per_instance=4),
docker_args="--shm-size=1g --ulimit memlock=-1",
environment_variables={"NCCL_DEBUG": "WARN"},#, "NCCL_IB_DISABLE": "0"},
tags={"project": "ssl-research", "job-purpose": "testing"}
)
job = ml_client.jobs.create_or_update(command_job)
print(f"Job created with ID: {job.id}")
Azure Machine Learning
Azure Machine Learning
An Azure machine learning service for building and deploying models.
3,338 questions
Sign in to answer