Logging in Azure ML

Question

I ran into a couple of issues while working with AzureML and thought I might get few insights from your side. The issues are from the following scenarios,

Logging in a Notebook Run: Working in AzureML Notebook can lead to losing experiment logs if the page is refreshed or accidently closed. The experiment continues running in the background, but the log freezes.

Logging Loss During Idle: If an experiment runs overnight and encounters an error, the compute instance may go idle and shut down (due to time limit), resulting in losing the experiment logs (e.g., terminal logs).
Based on the two scenarios above, my question would be: is it possible to recover the terminal logs natively in AzureML or do I need to manually create a log file for a notebook run to avoid losing the experiment logs? What is the ideal "long-term" setup for development to prevent/minimize the risk of losing experiments or error logs (crashes)? 

Answer 1

Hi there Ajam, Meraj

Thanks for using QandA platform

you can use Azure ML Logging & Experiment Tracking with mlflow. For notebook runs logging to Azure ML Runs makes usre logs are there even if the page is refreshed.

from azureml.core import Run  
run = Run.get_context()  
run.log("message", "Experiment started")  

For long-running exp avoid idle shutdown by setting Keep Alive policies in compute settings. also try, redirect logs to a persistent file using:

!nohup python script.py > output.log 2>&1 &  

You can store logs in Azure Blob Storage to track errors even after crashes.

If this helps kindly accept the answer thanks much.