HD Insights - converting the Aggregated Logs output into ADLSGen2 into Text format

Brendan Cassidy 26 Microsoft Employee

Can someone share a reference to systematically output logs from the HDinsights application runs into a readable format -- for the stdout. Today when running a Spark application through HdInsights (triggered by Livy API), after the application finishes the logs are accessible in HTML Format at location of https://<cluster-name>.azurehdinsights.net/yarnui/jobhistory/logs/{clusteruri}/port/30050/{container}/livy.

In addition, the logs are saved in the default storage location at /app-logs/livy/logs/{appid}. However these aggregated logs are encrypted in a binary format and are unable to be read. Is there a way to save these outputs in textual format or access the response via API? This would allow for systematically adding logs to failed applications and eliminate need to manually load the HTML response to find a stack trace.

MartinJaffer-MSFT 26,021 Reputation points

2020-10-22T21:41:25.927+00:00
Hello @Brendan Cassidy and welcome to Microsoft Q&A.

There are a couple features that might help you.

One of them is the Spark History Server, where you can download logs.

The second is the command :

yarn logs -applicationId <app ID>
Brendan Cassidy 26 Reputation points Microsoft Employee

2020-10-23T17:09:04.253+00:00

Hi @MartinJaffer-MSFT , thank you for the links.

Is there any way to automate the process of this? I know it is easy to view these logs inside the Spark History Server and/or the YARN Resource Manager UI. However, we were looking to see if there is a way to automatically download these logs into ADLs.

Example, run a spark job that fails due to a transform error, the stdout at the YARN Resource Manager will hold the stack trace and any outputted logs we have defined in the Job. Is there a way that these can be automatically stored in the ADLs in a .txt format? Or is the only way to do this to run the command on the Cluster and then manually copy that output to the adls?
MartinJaffer-MSFT 26,021 Reputation points

2020-10-23T21:19:17.71+00:00

I reached out internally, and was told, to get the aggregated logs, you have to do yarn logs -applicationId <Application ID> . While that is automatable, my instincts tell me other approaches ought to be possible.

For example, this Stack Overflow thread seems remarkable similar. They use the livy batch api.

Another thing worth trying, is making a call to the endpoint which is giving you html, but specify in the request headers you will only accept text, not text/html.
Brendan Cassidy 26 Reputation points Microsoft Employee

2020-10-23T22:01:02.217+00:00

Thank you for the help; Looks like I will go with one of the two solutions.
Brendan Cassidy 26 Reputation points Microsoft Employee

2020-11-11T18:23:27.957+00:00

Updates: If one would like to do this you can also set the YARN config directly through the Ambari or API Conf parameters.
https://hdinsight.github.io/yarn/empty-yarn-log.html