HDInsightSparkTaskEvents

Spark Task Events.

Table attributes

Attribute Value
Resource types microsoft.hdinsight/clusters
Categories Azure Resources
Solutions LogManagement
Basic log No
Ingestion-time transformation Yes
Sample Queries -

Columns

Column Type Description
ApplicationId string The application ID of the application producing the record.
AttemptId string The ID of task attempt.
_BilledSize real The record size in bytes
BytesRead long The number bytes read during the task.
BytesWritten long The number of bytes written by the task.
ClusterDnsName string The DNS name of the cluster running the task.
ClusterTenantId string The tenant ID of the cluster running the task.
DiskBytesSpilled long The number of disk bytes spilled.
EndReason string Reason why the task ended.
ExecutorCPUTime long The CPU time consumed by the task executor.
ExecutorDeserializeCPUTime long The CPU time the task executor spent deserializing.
ExecutorDeserializeTime long The time the task executor spent deserializing.
ExecutorId string The ID executor.
ExecutorRunTime long The time task executor spent running.
Failed bool Boolean describing whether the task failed.
FinishTime datetime The time (UTC) the task finished.
Host string The FQDN of the host running the task.
InputMetrics long The metrics associated with the task input.
IpAddress string The IP Address of the node running the task.
_IsBillable string Specifies whether ingesting the data is billable. When _IsBillable is false ingestion isn't billed to your Azure account
JvmGcTime long The time the JVM spent garbage collecting.
Killed bool Boolean describing whether the task was killed.
LaunchTime datetime The time (UTC) the task was launched.
MemoryBytesSpilled long The bytes of memory spilled.
NumUpdatedBlockStatuses long The number updated block statuses during the task.
OutputMetrics long The metrics associated with the task output.
PeakExecutionMemory long The peak amount of memory used during execution.
RecordsRead long The number of records read during the task.
RecordsWritten long The number of records written by the task.
Region string The region of the cluster running the task.
_ResourceId string A unique identifier for the resource that the record is associated with
ResultSerializationTime long The serialization time spent while getting the result.
ResultSize long Size of the result.
Role string The type of node running the task.
SchedulerDelay long The amount of delay the scheduler experienced.
ShuffleBytesWritten long The bytes written during the shuffle task.
ShuffleFetchWaitTime long The time spent waitng for fetching.
ShuffleLocalBlocksFetched long The number of local blocks fethced during the shuffle task.
ShuffleReadMetrics long The metrics associated with shuffle reads.
ShuffleRecordsWritten long The number of records written during the shuffle task.
ShuffleRemoteBlocksFetched long The number of remote blocks fethced during the shuffle task.
ShuffleTotalBlocksFetched long The number of blocks fethced during the shuffle task.
ShuffleTotalBytesRead long The number bytes read during the shuffle task.
ShuffleWriteMetrics long The metrics associated with shuffle writes.
ShuffleWriteTime long The time spent writing during the shuffle task.
SourceSystem string The type of agent the event was collected by. For example, OpsManager for Windows agent, either direct connect or Operations Manager, Linux for all Linux agents, or Azure for Azure Diagnostics
StageId string The ID of the stage associated with the task.
_SubscriptionId string A unique identifier for the subscription that the record is associated with
TaskId string The ID of the task.
TaskType string The task type.
TenantId string The Log Analytics workspace ID
TimeGenerated datetime The timestamp (UTC) of when the log was generated.
Type string The name of the table
UpdatedBlocks long The number of updated blocks.
UserSubscriptionId string The subscription ID of the cluster running the task