HDInsightSparkLogs

All logs from related to Spark including Jupyter and Livy logs.

Table attributes

Attribute Value
Resource types microsoft.hdinsight/clusters
Categories Azure Resources
Solutions LogManagement
Basic log No
Ingestion-time transformation Yes
Sample Queries -

Columns

Column Type Description
_BilledSize real The record size in bytes
ClusterName string Name of cluster.
CorrelationId string The ID for correlated events. Can be used to identify correlated events between multiple tables.
HostName string Name of host where log was emitted.
_IsBillable string Specifies whether ingesting the data is billable. When _IsBillable is false ingestion isn't billed to your Azure account
LogLevel string log level of message (INFO, WARN, ERROR, etc.).
LogType string The name of the log file that a record came from (e.g. SparkExecutorLog, SparkDriverLog).
Message string message from HBase log.
OperationName string The operation associated with log record.
_ResourceId string A unique identifier for the resource that the record is associated with
SourceSystem string The type of agent the event was collected by. For example, OpsManager for Windows agent, either direct connect or Operations Manager, Linux for all Linux agents, or Azure for Azure Diagnostics
_SubscriptionId string A unique identifier for the subscription that the record is associated with
TenantId string The Log Analytics workspace ID
TimeGenerated datetime The timestamp (UTC) of when the log was generated.
Type string The name of the table