Are task logs deleted when the node is deleted due to autoscaling?

Question

Hello,

I am trying to access the logs (stdout.txt and stderr.txt) of the tasks run.

As soon as the task is completed I lose access to those files. Looking at the browser console, if I search a file in the path field I get a 404 error, NodeNotFound, The specified node does not exist.

Is it caused because the node is deleted when the task completes? Is there a way to retain those files after the node is deleted?

Answer 1

Hello @Juan Manuel Pérez [Modelical]

That's right. When a node is removed from the pool, all of the files that are stored on the node are also removed. Because of this behavior, you should move the output of your task off of the node it's running on, and to a durable store before it completes. Similarly, if a task fails, you should move logs required to diagnose the failure to a durable store.

Batch has integrated support for Azure Storage to upload data via OutputFiles, and with various shared file systems, or you can perform the upload yourself in your tasks.

Regarding your question, you can save the log files by using Task output files. Task output files are a convenient way to save log files to Azure Storage. You can specify the output files in the task definition, and Batch will automatically upload the files to your storage account when the task completes.

Reference:

https://learn.microsoft.com/en-us/azure/batch/batch-task-output