Azure ML Studio Notebook - Kernel disconnects after running cell for about 10 minutes

André Zazzera 0

Hi,

I am running rather computationally expensive code in a Notebook in Azure ML Studio. My cells will queue, run, and then continue to the next. However, when I get to a cell with very complex code, the cell will run for about 10 minutes, and then the kernel disconnects, forcing me to restart the kernel and start again. Can anyone please advise how to keep the code running? Thank you.

NOTE: My compute is running the whole time. This happens with Python3 (ipykernel), Python 3.8 - AzureML, and Python 3.10 - SDK v2 kernels

dupammi 8,035 Reputation points Microsoft Vendor

2023-10-18T09:23:02.7833333+00:00
Hi @André Zazzera ,

Thanks for reaching out to Microsoft Q&A.

It appears that you're experiencing kernel disconnects when running computationally expensive code in Azure ML Studio Notebook. I will be happy to assist you in debugging this.

It looks like mostly an environment issue. Here are few pointers that might help to debug at your end.

You can create a custom kernel with Python 3.x and use it from the dropdown for any of your notebooks by following some simple steps.

Use the terminal of your compute by clicking the open terminal option.

Run the following commands on the terminal using conda to create a python 3.x environment and creating a kernel which can be used with your notebook.
For demo purpose, I was creating here 3.8: conda create -n py38 python=3.8 conda activate py38 conda install pip conda install ipykernel python -m ipykernel install --user --name python38 --display-name "Python 38"

Stop and start your compute instance and load the drop down to see the new kernel. Stopping and starting the kernel is available from the notebook screen as seen in the first screen shot.

There is also a way to login to the instance CLI through ssh. If the SSH have been enabled while instance creation. This could also help in getting additional logs to check what was the issue. Adding a new environment will also work.

To summarize, if you are looking to create environments to train your models and infer then you will have to use the curated, user/system-managed environments in your experiments. More details here.
If you are looking to just use the notebooks, then you can setup a custom kernel or virtual environment to run an experiment locally, as seen in first screen shot.

Please give these a try and let us know.

I hope this helps.

Thanks!
dupammi 8,035 Reputation points Microsoft Vendor

2023-10-19T13:43:52.2066667+00:00

Hi @André Zazzera ,

Following up to see if the above suggestion was helpful.

Thank you!
André Zazzera 0 Reputation points

2023-10-19T14:08:52.59+00:00

@dupammi It was not. I know how to create virtual environments and new kernels, and how to start a kernel, as I have code running. The problem is the kernel disconnecting itself while the code is running.
dupammi 8,035 Reputation points Microsoft Vendor

2023-10-19T15:27:32.89+00:00

Hi @André Zazzera ,

I apologize for any inconvenience you've experienced. I am actively investigating the situation and will provide you with updates as I make progress.

In the meantime, to enhance the handling of resource-intensive workloads, I recommend exploring the possibility of using a more robust VM configuration. It would also be helpful to test an alternative VM to determine if the current VM is contributing to the issue.

Additionally, if you suspect that the issue is specific to a particular cell in the notebook, consider implementing logging within your code to capture more details about the exact piece of code that might be causing the kernel to throttle.

Kindly try these recommendations and you should be able to fix the kernel throttling.

Thank you!
dupammi 8,035 Reputation points Microsoft Vendor

2023-10-22T01:10:53.05+00:00

Hi @André Zazzera

Following up to see if you have tried my above suggestion. It will be difficult to repro from my side, without the complex code you were trying. It looks like mostly an environment issue.

In case if you have any resolution, please do share that same with the community as it can be helpful to others.

Thanks!

Share via

Azure ML Studio Notebook - Kernel disconnects after running cell for about 10 minutes