How to use an environment to run code directly from my workspaceworkingdirectory ?

Question

How to use an environment to run code directly from my workspaceworkingdirectory ?

elias.alberto 1

I have some code which runs fine on my local machine in a conda environment, but it's super slow (as expected, because I don't have a decent GPU).

I got to Azure ML Studio, got a subscription, made a workspace with all assets needed. Also made a compute cluster with a healthy amount of GPU.
Based on image mcr.microsoft.com/azureml/openmpi4.1.0-cuda11.1-cudnn8-ubuntu18.04 , I built an environment in Azure with the ancient Python 3.6.13, keras 2.1.6 and tensorflow 1.15.5 which are necessary to run my code, and the environment was built without errors.
I've managed to put all the data and code under my workspaceworkingdirectory. I even created a Data Asset that points directly to it.

If I were on my machine, I would now open Anaconda Prompt and do this:

cd myfolder/myscript
conda activate myenv
python script.py <a bunch of parameters>

However, in Azure ML Studio I can't do the same directly from the terminal (outside of the enviroment) because of the dependencies. I also can't run it from the environment I built, because it's not listed there. If I open a terminal and type conda env list, I get this:

base /anaconda
azureml_py310_sdkv2 /anaconda/envs/azureml_py310_sdkv2
azureml_py38 * /anaconda/envs/azureml_py38
azureml_py38_PT_TF /anaconda/envs/azureml_py38_PT_TF

As simple as this issue may seem, I've been struggling for days to get this to run on Azure ML, and I haven't progressed at all. If anyone could point me in the right direction, I'd be immensely grateful.

Edit: Let me clarify one thing: on the Azure ML Studio, if I click on "notebooks" and get the terminal I used to download my code and data to workspaceworkingdirectory, and use that same terminal to invoke conda, I can create an environment and run my code with the same commands I mentioned above, BUT; that environment can't access any GPU acceleration (probably the plain terminal feature is missing the CUDA files, which are present on the docker image used by Azure to generate the proper Conda environment as I initially described). Also that environment manually created at the terminal doesn't seem to be persistent, I had to rebuild it from scratch when I switched computing instances and I might even need to rebuild again every time I restart that compute instance (haven't tested it yet because it's irrelevant unless I get the CUDA acceleration to run)

So, my goal is to get the image-based environment to access the workspaceworkingdirectory and run code there with R+W permissions to write the output files, OR to get cuda working on this environment I built from scratch on the terminal. Also, I need to see the tensorboard in either case.

Ramr-msft 17,826 Reputation points

2022-12-19T12:22:14.95+00:00

@elias.alberto Thanks for the details. Are you using the Azure Machine Learning Python SDK v2? If yes, Here is the Document and sample for TensorFlow in Azure ML.

To run an AzureML job, you'll need an environment. An environment is the software runtime and libraries that you want installed on the compute where you’ll be training. It is similar to your python emvironment on your local machine. AzureML provides many curated or readymade environments which are useful for common training and inference scenarios. You can also create your own “custom” environments using a docker image, or a conda configuration

https://github.com/Azure/azureml-examples/blob/83c67ec408f10e2e07b3a2a3e648023caa09e112/sdk/python/jobs/single-step/tensorflow/mnist-distributed-horovod/tensorflow-mnist-distributed-horovod.ipynb
elias.alberto 1 Reputation point

2022-12-20T23:45:29.637+00:00

@Ramr-msft Thank you for your reply. :)

I've read the links you provided, but they apply for someone using the Python SDK. I'm not using the Python SDK (yet), everything I did was on the web interface for Azure ML Studio available on ml.azure.com . That's where I created the workspace, the compute instances and clusters, the environment, and also where I managed to put my code and data at the storage space named "workspaceworkingdirectory", and managed to run all of it (without GPU acceleration) from a compute instance.

I saw the readymade environments, but I could not directly use them because of the ancient versions I need for Python and TF. However, I successfully built a custom environment from the same docker that was used as basis to build those readymade environments. The name of the docker used as base is mentioned in my initial post.

My current struggle is to start up that environment I created. The interface at ml.azure.com should make things easier, but at the 3rd step of building the job I'm presented with confusing things to set up : choose code location (and it doesn't let me select the code already in my storage, only from the blob storage which is blank because I could never access), it asks me to set up my inputs and outputs separately, and this adds unnecessary complexity. If possible, all I wanted to do is to fire up that environment and run some simple Terminal commands with my user storage mounted as R+W, so it would read the data and the code and also write data to the same storage.
Ramr-msft 17,826 Reputation points

2022-12-28T15:34:57.747+00:00

Thanks for the details. I have forwardedd to the product team to check on this.
elias.alberto 1 Reputation point

2022-12-29T19:40:21.767+00:00

Thank you for the help, @ramr-mstf :) Would you have any other suggestion? I still couldn't solve the issue (and my thesis is already overdue, while storage costs are growing).

I know that Microsoft did the GUI (in the form of the ML Studio) to make ML easier and more accessible, and I chose Azure because usually Microsoft try to make the lives of users easier. But this time I'm a bit disappointed... it's frustrating that nobody knows how to use it and there doesn't seem to be clear instructions (I found a non-reproducible step in the first ML Studio tutorial, that I had to circumnavigate myself). I personally fail to see the point in adding so much unnecessary undocumented complication while offering no option to do the most basic and intuitive solution... Feels like the SDK is the only real way to get things done and the ML Studio is an afterthought.
elias.alberto 1 Reputation point

2023-01-16T03:00:15.98+00:00

@Ramr-msft sorry for the delay, it would be very helpful if you could help me to reach out support because I can't initiate a support request with my account. I believe I have found two bugs in the Azure ML Studio that prevent the creation of jobs. My user id is ******@stud.uni-obuda.hu Thanks in advance
elias.alberto 1 Reputation point

2023-01-16T03:02:28.0133333+00:00

(duplicated, can't find how to delete this comment)

Your answer

Ramr-msft 17,826 Reputation points

2022-12-19T12:22:14.95+00:00

@elias.alberto Thanks for the details. Are you using the Azure Machine Learning Python SDK v2? If yes, Here is the Document and sample for TensorFlow in Azure ML.

To run an AzureML job, you'll need an environment. An environment is the software runtime and libraries that you want installed on the compute where you’ll be training. It is similar to your python emvironment on your local machine. AzureML provides many curated or readymade environments which are useful for common training and inference scenarios. You can also create your own “custom” environments using a docker image, or a conda configuration

https://github.com/Azure/azureml-examples/blob/83c67ec408f10e2e07b3a2a3e648023caa09e112/sdk/python/jobs/single-step/tensorflow/mnist-distributed-horovod/tensorflow-mnist-distributed-horovod.ipynb
elias.alberto 1 Reputation point

2022-12-20T23:45:29.637+00:00

@Ramr-msft Thank you for your reply. :)

I've read the links you provided, but they apply for someone using the Python SDK. I'm not using the Python SDK (yet), everything I did was on the web interface for Azure ML Studio available on ml.azure.com . That's where I created the workspace, the compute instances and clusters, the environment, and also where I managed to put my code and data at the storage space named "workspaceworkingdirectory", and managed to run all of it (without GPU acceleration) from a compute instance.

I saw the readymade environments, but I could not directly use them because of the ancient versions I need for Python and TF. However, I successfully built a custom environment from the same docker that was used as basis to build those readymade environments. The name of the docker used as base is mentioned in my initial post.

My current struggle is to start up that environment I created. The interface at ml.azure.com should make things easier, but at the 3rd step of building the job I'm presented with confusing things to set up : choose code location (and it doesn't let me select the code already in my storage, only from the blob storage which is blank because I could never access), it asks me to set up my inputs and outputs separately, and this adds unnecessary complexity. If possible, all I wanted to do is to fire up that environment and run some simple Terminal commands with my user storage mounted as R+W, so it would read the data and the code and also write data to the same storage.
Ramr-msft 17,826 Reputation points

2022-12-28T15:34:57.747+00:00

Thanks for the details. I have forwardedd to the product team to check on this.
elias.alberto 1 Reputation point

2022-12-29T19:40:21.767+00:00

Thank you for the help, @ramr-mstf :) Would you have any other suggestion? I still couldn't solve the issue (and my thesis is already overdue, while storage costs are growing).

I know that Microsoft did the GUI (in the form of the ML Studio) to make ML easier and more accessible, and I chose Azure because usually Microsoft try to make the lives of users easier. But this time I'm a bit disappointed... it's frustrating that nobody knows how to use it and there doesn't seem to be clear instructions (I found a non-reproducible step in the first ML Studio tutorial, that I had to circumnavigate myself). I personally fail to see the point in adding so much unnecessary undocumented complication while offering no option to do the most basic and intuitive solution... Feels like the SDK is the only real way to get things done and the ML Studio is an afterthought.
elias.alberto 1 Reputation point

2023-01-16T03:00:15.98+00:00

@Ramr-msft sorry for the delay, it would be very helpful if you could help me to reach out support because I can't initiate a support request with my account. I believe I have found two bugs in the Azure ML Studio that prevent the creation of jobs. My user id is ******@stud.uni-obuda.hu Thanks in advance
elias.alberto 1 Reputation point

2023-01-16T03:02:28.0133333+00:00

(duplicated, can't find how to delete this comment)

Share via

How to use an environment to run code directly from my workspaceworkingdirectory ?

Your answer