Pipeline events system table reference

Important

This system table is in Beta.

This article is a reference for the pipeline_events system table, which records Lakeflow pipelines event log entries for pipelines in your account. Each row is an immutable event from the pipeline event log, capturing lifecycle transitions, flow progress, data quality metrics, errors, cluster resources, and other operational data across all pipelines and workspaces within a region.

Requirements

  • To access this system table, users must either:

Available pipeline event tables

The pipeline events system table lives in the lakeflow_pipeline_events_preview schema during Beta, and moves to the lakeflow schema at general availability:

Table Description Supports streaming Free retention period Includes global or regional data
pipeline_events (Beta) Records pipeline event log entries emitted by pipeline runs Yes 13 months Regional

Note

The schema is lakeflow_pipeline_events_preview during Beta. At general availability, the table moves to the lakeflow schema (the final table path will be system.lakeflow.pipeline_events). Queries written against the Beta schema must be updated when the table moves.

Detailed schema reference

Pipeline events table schema

The pipeline events table is append-only. Each row records a single event emitted by a pipeline update at the time it was emitted, and rows are never modified or deleted in place.

Which fields are populated on a row depends on the event type. error, update_id, and many origin.* sub-fields are set only on events where they apply, and the structure of the details field also varies by event_type.

Use this table to query historical pipeline activity, to build alerts on pipeline failures, and to correlate pipeline behavior with other Lakeflow system tables.

Table path: system.lakeflow_pipeline_events_preview.pipeline_events

Primary key: (account_id, pipeline_event_id)

Column name Data type Description Notes
account_id string The ID of the account this pipeline event belongs to
workspace_id string The ID of the workspace this pipeline event belongs to
pipeline_id string The ID of the pipeline that emitted the event
update_id string The ID of the pipeline update that emitted the event
pipeline_event_id string Globally unique identifier for the event
event_type string The type of event (for example, flow_progress, update_progress, create_update) See Event type values for the full set of values.
origin struct Contextual metadata about the origin of the event like cloud provider, region, pipeline type, table or flow names, and other identifiers See Origin struct fields.
message string Human-readable description of the event May be empty for some events.
level string Severity level of the event One of INFO, WARN, ERROR, METRICS. See Level values.
maturity_level string Stability of the event schema One of STABLE, EVOLVING, DEPRECATED. See Maturity level values.
error struct Error details. Populated only for events that carry error information See Error struct fields.
details variant Event-specific payload. The fields it contains depend on the event_type See Details field.
event_time timestamp The time the event was emitted by the pipeline Timezone recorded as +00:00 (UTC).

Origin struct fields

Sub-field Data type Description
cloud string Cloud provider (for example, AWS, AZURE, GCP)
region string Cloud provider region
org_id bigint Workspace organization ID
pipeline_type string The type of pipeline
pipeline_name string The user-supplied name of the pipeline
cluster_id string The compute cluster ID backing the pipeline update
maintenance_id string The ID of the maintenance update, if the event is from a maintenance run
dataset_name string The name of the dataset (table or view) the event refers to
sink_name string The name of the sink the event refers to
catalog_name string The Unity Catalog catalog name
schema_name string The Unity Catalog schema name
flow_id string The ID of the flow the event refers to
flow_name string The name of the flow the event refers to
batch_id bigint The micro-batch ID for streaming flows. bigint for compatibility
request_id string The request ID that initiated the action
materialization_name string The materialization name
operation_id string The operation ID
source_name string The name of the data source
uc_table_id string The Unity Catalog table ID
ingestion_source_type string The type of ingestion source (for example, SQL_SERVER, SALESFORCE)
ingestion_source_connection_name string The connection name for the ingestion source
ingestion_source_catalog_name string The source catalog name in the upstream system
ingestion_source_schema_name string The source schema name in the upstream system
ingestion_source_table_name string The source table name in the upstream system
ingestion_source_table_version string The source table version, where applicable

Error struct fields

Sub-field Data type Description
fatal boolean Whether the error caused the update to terminate
exceptions array<struct> Chain of exceptions associated with the error (root cause last)
exceptions[].sql_state string SQLSTATE code, if available
exceptions[].error_class string Databricks error class, if available

Details field

The details column is a VARIANT, and the fields it contains depend on the event_type. For the fields available under each event type, see Pipeline event log schema. Use the variant_get function or dot syntax to read nested values. See the example queries below for typical access patterns.

The event_type key wraps the payload. For example, a flow_progress event's metrics are at $.flow_progress.metrics, not $.metrics. Include the event-type key in every path.

-- Using variant_get (lets you cast to a specific type)
SELECT
  pipeline_id,
  event_time,
  variant_get(details, '$.flow_progress.status', 'STRING')                       AS flow_status,
  variant_get(details, '$.flow_progress.metrics.num_output_rows', 'BIGINT')      AS rows_written,
  variant_get(details, '$.flow_progress.metrics.backlog_bytes', 'BIGINT')        AS backlog_bytes
FROM
  system.lakeflow_pipeline_events_preview.pipeline_events
WHERE
  event_type = 'flow_progress'
  AND event_time >= current_timestamp() - INTERVAL 1 HOUR

-- Using dot syntax (returns VARIANT, cast when needed)
SELECT
  pipeline_id,
  event_time,
  details:flow_progress.status::STRING               AS flow_status,
  details:flow_progress.metrics.num_output_rows::BIGINT AS rows_written
FROM
  system.lakeflow_pipeline_events_preview.pipeline_events
WHERE
  event_type = 'flow_progress'
  AND event_time >= current_timestamp() - INTERVAL 1 HOUR

Example queries

-- Flow throughput for a specific pipeline
SELECT
  origin.flow_name,
  date_trunc('HOUR', event_time) AS hour,
  SUM(variant_get(details, '$.flow_progress.metrics.num_output_rows', 'BIGINT')) AS rows_written
FROM
  system.lakeflow_pipeline_events_preview.pipeline_events
WHERE
  pipeline_id = '<your-pipeline-id>'
  AND event_type = 'flow_progress'
  AND event_time >= current_timestamp() - INTERVAL 7 DAYS
GROUP BY
  origin.flow_name,
  date_trunc('HOUR', event_time)
ORDER BY
  hour DESC,
  rows_written DESC

-- The latest error for each pipeline that has errored in the last 7 days, with the outermost exception.
-- The exception chain is ordered with the root cause last, so read element -1 for the root cause.
-- On many errors only the first element carries error_class and sql_state.
SELECT
  workspace_id,
  pipeline_id,
  event_time,
  event_type,
  message,
  error.exceptions[0].error_class AS exception_error_class,
  error.exceptions[0].sql_state   AS exception_sql_state
FROM
  system.lakeflow_pipeline_events_preview.pipeline_events
WHERE
  level = 'ERROR'
  AND event_time >= current_timestamp() - INTERVAL 7 DAYS
QUALIFY
  ROW_NUMBER() OVER (PARTITION BY workspace_id, pipeline_id ORDER BY event_time DESC) = 1
ORDER BY
  event_time DESC

-- Data quality: failed expectations by dataset, per update, in the last 1 day
SELECT
  pipeline_id,
  update_id,
  origin.dataset_name,
  expectation.name AS expectation_name,
  SUM(expectation.failed_records) AS failed_records
FROM
  system.lakeflow_pipeline_events_preview.pipeline_events
  LATERAL VIEW explode(variant_get(details, '$.flow_progress.data_quality.expectations', 'ARRAY<STRUCT<name:STRING,dataset:STRING,passed_records:BIGINT,failed_records:BIGINT>>')) AS expectation
WHERE
  event_type = 'flow_progress'
  AND event_time >= current_timestamp() - INTERVAL 1 DAY
GROUP BY
  pipeline_id,
  update_id,
  origin.dataset_name,
  expectation.name
HAVING
  SUM(expectation.failed_records) > 0
ORDER BY
  failed_records DESC

Common join patterns

Join with the pipelines table to filter by pipeline name

The pipelines table is a slowly changing dimension (SCD2). Take the latest version of each pipeline before joining.

WITH latest_pipelines AS (
  SELECT *
  FROM system.lakeflow.pipelines
  QUALIFY ROW_NUMBER() OVER (PARTITION BY workspace_id, pipeline_id ORDER BY change_time DESC) = 1
)
SELECT
  p.name AS pipeline_name,
  e.event_time,
  e.event_type,
  e.level,
  e.message
FROM
  system.lakeflow_pipeline_events_preview.pipeline_events e
JOIN
  latest_pipelines p
  ON e.workspace_id = p.workspace_id
  AND e.pipeline_id = p.pipeline_id
WHERE
  e.level = 'ERROR'
  AND e.event_time >= current_timestamp() - INTERVAL 24 HOURS
ORDER BY
  e.event_time DESC

Join with pipeline_update_timeline on update_id

SELECT
  u.period_start_time AS update_start,
  u.period_end_time   AS update_end,
  e.event_time,
  e.event_type,
  e.level,
  e.message
FROM
  system.lakeflow.pipeline_update_timeline u
JOIN
  system.lakeflow_pipeline_events_preview.pipeline_events e
  ON e.update_id = u.update_id
WHERE
  u.pipeline_id = '<your-pipeline-id>'
  AND u.period_start_time >= current_timestamp() - INTERVAL 7 DAYS
ORDER BY
  u.period_start_time DESC,
  e.event_time ASC

Setting up alerts

You can build alerts on pipeline_events using Databricks SQL alerts. Write a SQL query against pipeline_events (optionally joined with other Lakeflow system tables), schedule it on a SQL warehouse, and configure a notification destination (email, Slack, webhook, PagerDuty).

A few useful starting points:

Alert when no events arrived for a pipeline in the last N minutes

Use this to detect stuck or silently failing pipelines.

-- Returns one row per pipeline that has not emitted any event in the last 30 minutes.
-- The alert can trigger when this query returns any rows.
SELECT
  workspace_id,
  pipeline_id,
  MAX(event_time) AS last_event_time
FROM
  system.lakeflow_pipeline_events_preview.pipeline_events
WHERE
  event_time >= current_timestamp() - INTERVAL 24 HOURS
GROUP BY
  workspace_id,
  pipeline_id
HAVING
  MAX(event_time) < current_timestamp() - INTERVAL 30 MINUTES

Alert when the backlog for a specific flow is too high

Backlog is reported on flow_progress events as backlog_bytes, and for file sources also as backlog_files. Trigger when the most recent reading crosses a threshold (for example, 100 MB of unprocessed work). Not every source reports every metric, so filter on the one your source populates.

-- Returns the most recent backlog reading per flow for a given pipeline.
-- The alert can trigger when backlog_bytes exceeds the threshold for any flow.
WITH latest_flow_progress AS (
  SELECT
    workspace_id,
    pipeline_id,
    origin.flow_name,
    event_time,
    CASE
      WHEN variant_get(details, '$.flow_progress.status', 'STRING') = 'COMPLETED' THEN 0
      ELSE variant_get(details, '$.flow_progress.metrics.backlog_bytes', 'BIGINT')
    END AS backlog_bytes
  FROM
    system.lakeflow_pipeline_events_preview.pipeline_events
  WHERE
    pipeline_id = '<your-pipeline-id>'
    AND event_type = 'flow_progress'
    AND event_time >= current_timestamp() - INTERVAL 1 HOUR
    AND (
      variant_get(details, '$.flow_progress.metrics.backlog_bytes', 'BIGINT') IS NOT NULL
      OR variant_get(details, '$.flow_progress.status', 'STRING') = 'COMPLETED'
    )
  QUALIFY
    ROW_NUMBER() OVER (PARTITION BY workspace_id, pipeline_id, origin.flow_name ORDER BY event_time DESC) = 1
)
SELECT *
FROM latest_flow_progress
WHERE backlog_bytes > 100000000

To see which source is behind, $.flow_progress.metrics.source_metrics is an array of per-source readings, each with source_name alongside that source's backlog_bytes, backlog_records or backlog_files.

Alert on data-quality drops in a pipeline

Each flow_progress event reports the number of rows dropped by EXPECT … DROP expectations. Sum these per dataset over an update window and alert when the total exceeds a threshold.

-- Returns datasets where more than 100 rows were dropped by expectations, per update, in the last hour.
SELECT
  pipeline_id,
  update_id,
  origin.dataset_name,
  SUM(variant_get(details, '$.flow_progress.data_quality.dropped_records', 'BIGINT')) AS dropped_records
FROM
  system.lakeflow_pipeline_events_preview.pipeline_events
WHERE
  event_type = 'flow_progress'
  AND event_time >= current_timestamp() - INTERVAL 1 HOUR
GROUP BY
  pipeline_id,
  update_id,
  origin.dataset_name
HAVING
  SUM(variant_get(details, '$.flow_progress.data_quality.dropped_records', 'BIGINT')) > 100

Alert when a flow processes too few rows

flow_progress events report metrics.num_output_rows as a per-micro-batch count, so summing the events in a window gives the rows written over that window. Create an alert for when throughput drops below an expected floor. For example, a flow that normally writes thousands of rows per hour but produces near zero can indicate a misconfigured source.

This query only reports flows that emitted a flow_progress event with a row count in the window. A fully stalled flow emits no events, so pair this alert with the missing-events alert above.

-- Returns flows that wrote fewer than 100 rows in the last hour.
-- The alert can trigger when this query returns any rows.
SELECT
  pipeline_id,
  origin.flow_name,
  SUM(variant_get(details, '$.flow_progress.metrics.num_output_rows', 'BIGINT')) AS rows_written_last_hour
FROM
  system.lakeflow_pipeline_events_preview.pipeline_events
WHERE
  pipeline_id = '<your-pipeline-id>'
  AND event_type = 'flow_progress'
  AND event_time >= current_timestamp() - INTERVAL 1 HOUR
  AND variant_get(details, '$.flow_progress.metrics.num_output_rows', 'BIGINT') IS NOT NULL
GROUP BY
  pipeline_id,
  origin.flow_name
HAVING
  SUM(variant_get(details, '$.flow_progress.metrics.num_output_rows', 'BIGINT')) < 100

Alert when new-data latency is too high

For streaming flows, flow_progress events report latency in streaming_metrics. stream_latency_ms is the time from when data landed upstream to when the micro-batch committed to the Delta table. You can set a trigger for when the most recent reading crosses a threshold (for example, 5 minutes).

Only streaming flows with a tagged event time report stream_latency_ms, and only when SDP time-metrics are enabled. Other flows return NULL on every event, and this alert never fires for them.

-- Returns the most recent new-data latency per flow for a given pipeline.
-- The alert can trigger when stream_latency_ms > 300000 (5 minutes) for any flow.
WITH latest_latency AS (
  SELECT
    workspace_id,
    pipeline_id,
    origin.flow_name,
    event_time,
    CASE
      WHEN variant_get(details, '$.flow_progress.status', 'STRING') = 'COMPLETED' THEN 0
      ELSE variant_get(details, '$.flow_progress.streaming_metrics.stream_latency_ms', 'BIGINT')
    END AS stream_latency_ms
  FROM
    system.lakeflow_pipeline_events_preview.pipeline_events
  WHERE
    pipeline_id = '<your-pipeline-id>'
    AND event_type = 'flow_progress'
    AND event_time >= current_timestamp() - INTERVAL 1 HOUR
    AND (
      variant_get(details, '$.flow_progress.streaming_metrics.stream_latency_ms', 'BIGINT') IS NOT NULL
      OR variant_get(details, '$.flow_progress.status', 'STRING') = 'COMPLETED'
    )
  QUALIFY
    ROW_NUMBER() OVER (PARTITION BY workspace_id, pipeline_id, origin.flow_name ORDER BY event_time DESC) = 1
)
SELECT *
FROM latest_latency
WHERE stream_latency_ms > 300000

Tips for production alerts

  • Filter by pipeline_id (and workspace_id if you maintain alerts per workspace) so each alert targets a specific scope rather than the whole account.
  • Choose an evaluation cadence that matches the alert's sensitivity. Use a short interval (for example, every 5 minutes) for fast-fail signals and a longer interval (for example, hourly) for backlog and data-quality trends. A "trigger when query returns more than 0 rows" condition works for most cases.

Reference values

Level values

Value Description
INFO Normal pipeline activity (flow progress, update lifecycle transitions, configuration changes).
WARN Non-fatal issues the pipeline recovered from, or that may require attention.
ERROR Failures that prevented the pipeline from making progress on a flow or update.
METRICS Quantitative measurements emitted during execution (row counts, throughput, latency).

Maturity level values

Value Description
STABLE The event schema is stable. Breaking changes are not expected. Safe to build production queries and alerts on.
EVOLVING The event schema may change in future releases. Use with care.
DEPRECATED The event type or schema is deprecated and will be removed in future releases. Migrate off it.

Event type values

The event_type field is an enumeration. The full set of values:

Value Description
create_update A new pipeline update was requested.
update_progress A pipeline update transitioned through a lifecycle state.
flow_progress A flow (dataset) within an update transitioned through a state.
flow_definition Static metadata about a flow.
dataset_definition Static metadata about a dataset.
sink_definition Static metadata about an output sink.
deprecation A deprecated feature was used by the pipeline.
autoscale Cluster autoscaling decision.
unsupported_operation An operation that is not supported in the current configuration.
cluster_resources Task slot and autoscale metrics for the backing compute.
planning_information Planning-phase information for the update.
gc_pressure Garbage collection pressure on driver or executors.
abnormal_termination The update terminated abnormally.
disk_space Disk space pressure on the cluster.
hook_progress Lifecycle progress of a pipeline hook.
dataset_life_cycle A dataset lifecycle event.
background_operation A background operation transitioned through a state.
remote_api_usage The pipeline made an outbound API call.
operation_progress Progress for a generic operation.
stream_progress Progress for a streaming query backing a flow.
rewind_summary Summary of a pipeline rewind operation.
advisory An advisory message from the engine.
runtime_details Detailed runtime configuration.
resource_info Resource information (cluster, instance type, etc.).
file_notification_set_up File notification setup status (for cloud-files sources).
behavior_change_in_spark_connect Behavior-change notification under Spark Connect.
user_action A user-initiated action against the pipeline.
user_code_context Context about the user code associated with the event.
  • Last updated on