I have been testing Azure ML experiments running locally on my machine with docker. So far I have run into the same issue using several curated environments as well as using a conda dependencies file. The run job is submitted successfully Docker container builds successfully The docker run command fails due to the --gpu all switch This switch gets added to every docker container I try to launch locally (doesn't matter the container type). I have tested this a few different ways Using VScode and VScode insiders Running the experiment from code Running Experiment the Azure ML Extension in VScode All attempts end with the docker container failing to run. Any thought on how to fix this would be appreciated.

Docker container fails to run due to invalid --GPU switch

Kevin Oliver 1

I have been testing Azure ML experiments running locally on my machine with docker. So far I have run into the same issue using several curated environments as well as using a conda dependencies file.

The run job is submitted successfully
Docker container builds successfully
The docker run command fails due to the --gpu all switch

This switch gets added to every docker container I try to launch locally (doesn't matter the container type).

I have tested this a few different ways

Using VScode and VScode insiders
Running the experiment from code
Running Experiment the Azure ML Extension in VScode

All attempts end with the docker container failing to run.

Any thought on how to fix this would be appreciated.

romungi-MSFT 42,791 Reputation points Microsoft Employee

2021-01-11T07:48:28.177+00:00

@Kevin Oliver It would be great if you could share any documentation link that you are using to run your container to check and replicate the issue. There are AzureML base images for GPU and CPU on docker hub, I wonder if you are using a GPU image that might be causing it to fail.
Kevin Oliver 1 Reputation point

2021-01-11T14:02:14.387+00:00

@romungi-MSFT I have used several guides, but this one specifically has instructions for running a container locally

https://github.com/Azure/MachineLearningNotebooks/blob/3adebd11278686a23c13434b42340acb248b3133/configuration.ipynb
romungi-MSFT 42,791 Reputation points Microsoft Employee

2021-01-12T08:08:46.807+00:00

The steps in the configuration notebook file creates a compute cluster either a CPU or GPU for the attached workspace on Azure. You can use this sample to train on local for a simple experiment. With respect to the error though it would be great if you could detail the steps so it can be replicated to check what could be incorrect. Thanks!!
Kevin Oliver 1 Reputation point

2021-01-12T21:29:40.103+00:00
@romungi-MSFT

Using the sample you suggested, here are the steps I go through to get the error.

Run Notebook step successfully till you hit Section 6.C.c

Either the Run or run.wait_for_completion(show_output=True) cells will fail with the same error
The results are attached in the docker_output file.
55931-docker-output.txt

Docker Error
The error received during the docker run is attached as well.
Sample
[2021-01-12T21:03:34.495116] Logging experiment running status in history service.
Running: 'docker', 'run', '--name', 'train-on-local_1610485398_90bae222', '--rm', '-v', 'C:\Users\kevba\AppData\Local\Temp\azureml_runs\train-on-local_1610485398_90bae222:/azureml-run', '--shm-size', '2g', '--gpus', 'all', '-e', ...
55932-run-local-docker-fail.txt
romungi-MSFT 42,791 Reputation points Microsoft Employee

2021-01-13T06:20:37.977+00:00

@Kevin Oliver you seem to be running on windows. Does your kernel already run on a docker container? The error as you pointed out is with a GPU capability which will not work with this sample on local as mentioned in the notebook. Are you using a user managed environment or a system managed environment?
Kevin Oliver 1 Reputation point

2021-01-13T14:41:03.92+00:00

I have been using a user-managed environment.

My kernel does not run in a container

PC Setup-
Windows 10 pro - Insiders build
WSL install
Docker Desktop installed on windows.
If i use the docker run command that is generated without the --gpu switch, the docker container builds successfully. I am just trying to understand where in the environment creation steps it is choosing to add the GPU switch. Since CPU docker container is the only thing that will run on windows right now, even selecting a Docker GPU image should fail will before the container creation.
romungi-MSFT 42,791 Reputation points Microsoft Employee

2021-01-15T12:28:26.463+00:00

@Kevin Oliver The curated environment name determines how your environment is setup in this case. If you are only running step 6.c it should run the CPU docker image intelmpi2018.3-ubuntu16.04
I think the script config might be overwritten in your case from earlier 6.x steps which might be using the configuration for GPU setup.

Share via

Docker container fails to run due to invalid --GPU switch