Telemetry table reference

Important

This feature is in Beta. Workspace admins can control access to this feature from the Previews page. See Manage Azure Databricks previews.

This page documents the tables Lakebase writes to your Unity Catalog schema once you configure observability for a project. These are the tables described in How it works. Insights and Genie read this same data, and because it lands in your own Unity Catalog, you can query it directly with any Azure Databricks tool.

Note

This is a Beta feature. Table and column names might change before general availability.

Columns common to every table

Every table below includes these columns, which identify where a row came from:

Column Type Description
project_id STRING The Lakebase project the row came from.
branch_id STRING The branch the row came from.
endpoint_id STRING The compute endpoint the row came from.
compute_id STRING The specific compute instance. This changes when a compute scales to zero and resumes.
ts TIMESTAMP When the row was recorded.
compute_mode STRING The compute's mode at the time, for example read-write or read-only.

Tables with names ending in _counters also include a previous_ts column, marking the start of the interval a row's values accumulated over. These are cumulative values for that interval, not point-in-time readings. Tables ending in _gauges are point-in-time readings instead, and have no previous_ts.

The tables below list only the columns beyond these common ones.

Session activity

A snapshot of every active backend, sampled continuously. This is the same pattern as AWS Performance Insights or Oracle's Active Session History (ASH): if a session is doing anything, including waiting, it shows up here.

active_session_history

Column Type Description
sample_seq LONG Ordinal position of this sample within its batch.
pid LONG The backend's process ID.
datid LONG The database's object ID.
userid LONG The connecting role's object ID.
queryid LONG Identifies the query being run. Matches queryid in pg_stat_statements_counters.
wait_event_info LONG Internal encoding of the wait event.
wait_event_type STRING The wait event's category, for example Timeout or Lock.
wait_event STRING The specific wait event, for example PgSleep.
backend_state LONG The backend's state, for example active or idle.
backend_type LONG The type of backend process.
xact_age_ms LONG How long the current transaction has been open.
query_age_ms LONG How long the current query has been running.
blocking_pid LONG The PID of a session blocking this one, if any.
leader_pid LONG The PID of the parallel query leader, if this backend is a parallel worker.
flags LONG Internal status flags.
plan_fingerprint LONG Links to a specific plan shape in plan_history.
appname_hash LONG A hash of the connecting application's name.

Wait events

Aggregated wait-event counts and time, complementing the per-sample view in active_session_history.

wait_event_counters

Column Type Description
wait_event_id LONG Internal encoding of the wait event.
wait_class_name STRING The wait event's category.
wait_event_name STRING The specific wait event.
wait_event_count LONG How many times this wait event occurred in the interval.
wait_event_time DOUBLE Total time spent in this wait event during the interval.

Query plans

A per-execution record of query plans, including the full plan and Neon-specific storage stats. This is the most detailed table, and typically where a slow-query investigation ends up.

plan_history

Column Type Description
row_position LONG Ordinal position of this row within its batch.
analyze_enabled BOOLEAN Whether the plan includes actual execution statistics (EXPLAIN ANALYZE), not just estimates.
buffers_enabled BOOLEAN Whether the plan includes buffer usage statistics.
query_hash STRING Hash identifying the query text.
queryid LONG Identifies the query. Matches queryid in pg_stat_statements_counters and active_session_history.
duration_ms DOUBLE How long the query took to run.
total_cost DOUBLE The planner's estimated cost for the query.
plan_rows LONG The planner's estimated row count.
actual_rows LONG The actual row count returned.
plan_fingerprint LONG Identifies this specific plan shape. Matches plan_fingerprint in active_session_history.
lock_wait_time_ms DOUBLE Time spent waiting on locks.
planning_time_ms DOUBLE Time spent planning the query, separate from execution.
shared_blks_hit / shared_blks_read / shared_blks_dirtied / shared_blks_written LONG Shared buffer cache activity for this execution.
local_blks_hit / local_blks_read / local_blks_dirtied / local_blks_written LONG Local buffer activity, for temporary tables.
temp_blks_read / temp_blks_written LONG Temporary file activity, for example from a large sort or hash.
neon_getpage_count LONG Number of page requests sent to the pageserver.
neon_file_cache_hits LONG Number of pages served from the local file cache instead of the pageserver.
neon_getpage_wait_us LONG Time spent waiting on pageserver page requests, in microseconds.
backend_pid LONG The backend process ID that ran this query.
lock_wait_count LONG Number of times this execution waited on a lock.
userid / dbid LONG The connecting role's and database's object IDs.
query_text STRING The query text.
plan_json STRING The full query plan, as JSON.
usename / datname STRING The connecting role's and database's names.
application_name / client_addr STRING The connecting application's name and client address.
wait_events STRING Wait events observed during this execution.
parameters STRING Bind parameters used in the query, if any.
blocking_pids STRING PIDs of sessions that blocked this execution, if any.
neon_branch_id / neon_endpoint_id / neon_timeline_id STRING Internal Neon identifiers for the branch, endpoint, and timeline.
cpu_user_time_ms / cpu_sys_time_ms DOUBLE CPU time consumed by this execution, split into user and system time.
voluntary_csw / involuntary_csw LONG Context switches during this execution.
trace_id / span_id / trace_flags / service_name STRING / STRING / LONG / STRING OpenTelemetry-style tracing identifiers, if the query was part of a traced request.
timeline_id LONG Internal timeline identifier.
truncated BOOLEAN Whether this row's data (for example, a large plan) was truncated before being written.

Query statistics

Aggregated per-query statistics, sourced from the standard pg_stat_statements extension.

pg_stat_statements_counters

Column Type Description
userid / dbid LONG The connecting role's and database's object IDs.
queryid LONG Identifies the query. Matches queryid in plan_history and active_session_history.
toplevel BOOLEAN Whether this row is for a top-level statement, as opposed to one nested inside a PL/pgSQL function or procedure. See the PL/pgSQL limitation.
query STRING The query text.
calls LONG Number of times this query was executed in the interval.
plans LONG Number of times this query was planned in the interval.
rows LONG Total rows returned or affected.
total_exec_time / total_plan_time DOUBLE Total time spent executing and planning this query in the interval.
shared_blks_hit / shared_blks_read / shared_blks_dirtied / shared_blks_written LONG Shared buffer cache activity.
local_blks_hit / local_blks_read / local_blks_dirtied / local_blks_written LONG Local buffer activity, for temporary tables.
temp_blks_read / temp_blks_written LONG Temporary file activity.
wal_bytes / wal_fpi / wal_records LONG Write-ahead log volume generated by this query.
jit_emission_count / jit_emission_time / jit_functions / jit_generation_time / jit_inlining_count / jit_inlining_time / jit_optimization_count / jit_optimization_time Mixed JIT (just-in-time) compilation statistics, if JIT was used for this query.

Schema changes

Schema changes: what changed, when, and by whom. This is what Genie reads to trace an incident back to a schema change, whether from a deploy or a manual edit.

ddl_history

Column Type Description
row_position LONG Ordinal position of this row within its batch.
command_tag STRING The type of DDL command, for example ALTER TABLE.
object_type / object_name STRING The kind and name of the object that changed.
schema_name STRING The Postgres schema the object belongs to.
query_text STRING The full DDL statement.
duration_ms DOUBLE How long the DDL statement took to run.
userid / usename LONG / STRING The connecting role's object ID and name.
application_name STRING The connecting application's name.
backend_pid LONG The backend process ID that ran the statement.
dbid LONG The database's object ID.
search_path STRING The session's search path at the time.
ddl_json STRING A structured, parsed representation of the DDL statement.

Note

DDL capture has a real gap: a change made right around a compute resuming from scale-to-zero can be missed. If a schema change doesn't appear here even though you know it happened, that's the most likely reason. See Limitations.

Postgres logs

Raw Postgres log messages, including the specific error behind a failure.

postgres_logs

Column Type Description
sample_seq LONG Ordinal position of this row within its batch.
elevel LONG Internal log level encoding.
severity STRING The log severity, for example ERROR or LOG.
sqlstate STRING The Postgres SQLSTATE error code, for example 42703 for an undefined column.
backend_pid / leader_backend_pid LONG The backend process ID, and its parallel query leader's PID if applicable.
userid / usename LONG / STRING The connecting role's object ID and name.
datid / datname LONG / STRING The database's object ID and name.
application_name / client_addr STRING The connecting application's name and client address.
schema_name / table_name / column_name / constraint_name / datatype_name STRING The schema, table, column, constraint, or data type referenced by the error, when Postgres has one.
funcname STRING The function the error occurred in, if applicable.
filename / lineno STRING / LONG The Postgres source file and line the error originated from.
message STRING The log message text.
detail / hint / context STRING Additional detail, a suggested fix, and the error's context, when Postgres has them.
internalquery STRING An internally generated query related to the error, if applicable.
statement STRING The statement that triggered this log entry.
truncated_mask LONG Indicates which fields, if any, were truncated before being written.

Note

The structured schema_name, table_name, and column_name fields aren't populated for every error type. A plain "column does not exist" parse error (SQLSTATE 42703), for example, leaves them empty even though message names the column directly. Don't rely on these fields being populated. Check message first.

Compute metrics

Compute-level resource usage: CPU, memory, disk and network I/O, and the local file cache (LFC), Lakebase's compute-side cache in front of the pageserver.

compute_counters

Column Type Description
backpressure_throttling_seconds DOUBLE Time the storage layer spent throttling writes in this interval. A non-zero value means write volume exceeded what storage can absorb.
host_busiest_cpu_seconds / host_cpu_seconds DOUBLE CPU time consumed, for the busiest core and in total.
host_disk_read_bytes / host_disk_written_bytes DOUBLE Disk I/O volume.
host_network_receive_bytes / host_network_transmit_bytes DOUBLE Network I/O volume.
lfc_evictions / lfc_hits / lfc_misses / lfc_writes LONG Local file cache activity for this interval.
replica_lfc_redo_evictions LONG Local file cache evictions caused by replica redo, if this compute is a read replica.

compute_gauges

Column Type Description
current_lsn LONG The current write-ahead log position.
replica_wal_replay_lsn / replica_wal_receive_lsn LONG WAL replay and receive positions, if this compute is a read replica.
max_connections LONG The configured connection limit.
lfc_working_set_size_5m / lfc_working_set_size_15m / lfc_working_set_size_60m LONG Estimated working set size over the last 5, 15, and 60 minutes.
lfc_size_limit / lfc_allocated / lfc_used LONG Local file cache capacity and current usage.
total_size_limit / total_size LONG Overall storage capacity and current usage.
replication_lag_bytes / replication_lag_time_ms LONG / DOUBLE Replication lag, if this compute is a read replica.
host_logical_cpus / host_memory_total_bytes DOUBLE The compute's provisioned CPU and memory.

Database metrics

Per-database activity: rows changed, transactions, sessions, and vacuum health. This is where table bloat and autovacuum signals live.

database_counters

Column Type Description
datid / datname LONG / STRING The database's object ID and name.
active_time_ms / idle_in_transaction_time_ms / session_time_ms DOUBLE Time spent active, idle in an open transaction, and connected overall.
deadlocks LONG Number of deadlocks detected.
rows_deleted / rows_fetched / rows_inserted / rows_returned / rows_updated LONG Row-level activity for the interval.
sessions_abandoned / sessions_established / sessions_fatal / sessions_killed LONG Session lifecycle counts.
shared_blks_hit / shared_blks_read LONG Shared buffer cache activity.
xact_commit / xact_rollback LONG Committed and rolled-back transactions.

database_gauges

Column Type Description
datid / datname LONG / STRING The database's object ID and name.
numbackends LONG Current number of connections to this database.
size LONG The database's on-disk size.
oldest_frozen_xid_age / oldest_mxid_age LONG Transaction ID and multixact ID age of the oldest unfrozen row. Rising values are the earliest signal of vacuum falling behind, well before bloat becomes visible in table size.

Next steps

  • Last updated on