How to persist stdout and stderr files generated by Azure Batch through the Azure Portal?

Question

How to persist stdout and stderr files generated by Azure Batch through the Azure Portal?

Carlos Santos 0

I have a few jobs on Azure Batch that have been presenting some failures lately. However, it is not possible to diagnose what is the problem, since the stderr and stdout files are deleted right after the task has been executed. I read that this has to do with the fact that the jobs are not saving these files in the Storage Account container, so they are only available as long as the node is alive (a few seconds after the execution from what I tested).

Is there a way to set the jobs to save these files somewhere just so I can understand what happens when my application throws an exception? It has to be through Azure Portal, since I don't have permission to install any Azure-related software right now in my company.

VasimTamboli 5,550 Reputation points MVP

2023-06-02T15:37:21.6866667+00:00

You can configure your Batch job to store these files in a storage account. Here's how you can do it:

Create a Storage Account: If you don't already have a storage account, create one in the Azure Portal. Make note of the storage account name and the access key.

Update the Batch Job: In the Azure Portal, navigate to your Batch account and select the appropriate job you want to configure. Go to the "Tasks" tab and select the specific task within the job.

Configure Task Output Files: In the task configuration, you can specify the output files you want to persist. In this case, you would specify the stdout and stderr files. Set the stdoutStderrPathPrefix property to a path within the storage account where you want the files to be stored. For example, you could use something like stdout-stderr-container/stdoutstderr.

Specify Storage Account Details: In the same task configuration, you need to provide the details of the storage account you created. Set the storageAccountSettings property with the storage account name and access key.

Save and Submit the Job: Save the task configuration and submit the job. The stdout and stderr files generated by the task will now be stored in the specified storage account container.

By following these steps, you can ensure that the stdout and stderr files are saved in a storage account and accessible even after the task has finished executing. This allows you to analyze any exceptions or failures that occurred during the job execution.

Please note that this configuration needs to be done at the task level for each individual task within your job. You can repeat these steps for other tasks to persist their stdout and stderr files as well.

If you have any further questions or need additional assistance, please let me know.
Carlos Santos 0 Reputation points

2023-06-06T16:44:54.4166667+00:00

After selecting the task, there is no "Output Files" option, as you can see below:

The only option here that might resemble what you said are these "Saved output files" and "Saved logs", both of which are empty:

What should I do?
Bailey 15 Reputation points

2023-11-30T15:14:00.5633333+00:00

It's November 2023 and I have the same problem.

I see a pattern with all these "no-code" solutions offered through the Azure portal, every product I try to use is half-baked, missing features or things not working as advertised. AI studio, data factory, batch. you name it.

Do people actually use Azure in production?