Lakebase telemetry in system tables

Important

This feature is in Beta. To use it, a workspace admin must turn on Lakebase Observability in System Tables from the Previews page. See Manage Azure Databricks previews.

Lakebase captures its observability telemetry into the system.lakebase schema, starting with advanced Postgres signals. Capture is automatic. With that preview enabled, telemetry lands in these tables while a project's compute runs. There is no configuration to create, no destination to choose, and no credentials to provide. Insights and Genie read the same tables, and you can query them with any Azure Databricks tool.

Note

Table and column names might change before general availability.

Requirements

  • The Lakebase Observability in System Tables preview enabled for your workspace.
  • A Lakebase project on AWS or Azure. See Get a Postgres database.

What gets captured

Lakebase records the signals you need to explain a slow or failing database, across the following tables:

What's captured Table name
Active sessions and wait events system.lakebase.active_session_history, system.lakebase.wait_event_counters
Query plans and per-query execution stats system.lakebase.plan_history, system.lakebase.pg_stat_statements_counters
Schema changes: what changed, when, and by whom system.lakebase.ddl_history
Database statistics: size, rows read and modified, and session counts system.lakebase.database_counters, system.lakebase.database_gauges
Compute resource usage: CPU, memory, cache, connections system.lakebase.compute_counters, system.lakebase.compute_gauges
Raw Postgres log messages, including the error behind a failure system.lakebase.postgres_logs

Records are retained for 7 days. The rest of this page documents every column in each table. For where these tables sit among the other system tables, see Lakebase system table reference.

Access

The system.lakebase tables are governed by Unity Catalog. Users with both the account admin and metastore admin roles can read the telemetry and manage access to it. See Manage privileges in Unity Catalog.

Access applies at the schema level, so a user who can read system.lakebase sees telemetry for every project in the account.

Masked statement text

Columns that hold statement text return <REDACTED> unless you are an account admin or a member of the databricks_pii_access account-level group. An account admin creates and manages that group. See Create and manage the databricks_pii_access group and Access masked statement text.

Columns common to every table

Every table below includes these columns, which identify where a row came from:

Column Type Description
project_id STRING The Lakebase project the row came from.
branch_id STRING The branch the row came from.
endpoint_id STRING The compute endpoint the row came from.
compute_id STRING The specific compute instance. This changes when a compute scales to zero and resumes.
ts TIMESTAMP When the row was recorded.
compute_mode STRING The compute's mode at the time, for example read-write or read-only.

Tables with names ending in _counters also include a previous_ts column, marking the start of the interval a row's values accumulated over. These are cumulative values for that interval, not point-in-time readings. Tables ending in _gauges are point-in-time readings instead, and have no previous_ts.

The tables below list only the columns beyond these common ones.

Session activity

A snapshot of every active backend, sampled continuously. This is the same pattern as AWS Performance Insights or Oracle's Active Session History (ASH): if a session is doing anything, including waiting, it shows up here.

active_session_history

Column Type Description
sample_seq LONG Ordinal position of this sample within its batch.
pid LONG The backend's process ID.
datid LONG The database's object ID.
userid LONG The connecting role's object ID.
queryid LONG Identifies the query being run. Matches queryid in pg_stat_statements_counters.
wait_event_info LONG Internal encoding of the wait event.
wait_event_type STRING The wait event's category, for example Timeout or Lock.
wait_event STRING The specific wait event, for example PgSleep.
backend_state LONG The backend's state, for example active or idle.
backend_type LONG The type of backend process.
xact_age_ms LONG How long the current transaction has been open.
query_age_ms LONG How long the current query has been running.
blocking_pid LONG The PID of a session blocking this one, if any.
leader_pid LONG The PID of the parallel query leader, if this backend is a parallel worker.
flags LONG Internal status flags.
plan_fingerprint LONG Links to a specific plan shape in plan_history.
appname_hash LONG A hash of the connecting application's name.

Wait events

Aggregated wait-event counts and time, complementing the per-sample view in active_session_history.

wait_event_counters

Column Type Description
wait_event_id LONG Internal encoding of the wait event.
wait_class_name STRING The wait event's category.
wait_event_name STRING The specific wait event.
wait_event_count LONG How many times this wait event occurred in the interval.
wait_event_time DOUBLE Total time spent in this wait event during the interval.

Query plans

A per-execution record of query plans, including the full plan and Neon-specific storage stats. This is the most detailed table, and typically where a slow-query investigation ends up.

plan_history

Column Type Description
row_position LONG Ordinal position of this row within its batch.
analyze_enabled BOOLEAN Whether the plan includes actual execution statistics (EXPLAIN ANALYZE), not just estimates.
buffers_enabled BOOLEAN Whether the plan includes buffer usage statistics.
query_hash STRING Hash identifying the query text.
queryid LONG Identifies the query. Matches queryid in pg_stat_statements_counters and active_session_history.
duration_ms DOUBLE How long the query took to run.
total_cost DOUBLE The planner's estimated cost for the query.
plan_rows LONG The planner's estimated row count.
actual_rows LONG The actual row count returned.
plan_fingerprint LONG Identifies this specific plan shape. Matches plan_fingerprint in active_session_history.
lock_wait_time_ms DOUBLE Time spent waiting on locks.
planning_time_ms DOUBLE Time spent planning the query, separate from execution.
shared_blks_hit / shared_blks_read / shared_blks_dirtied / shared_blks_written LONG Shared buffer cache activity for this execution.
local_blks_hit / local_blks_read / local_blks_dirtied / local_blks_written LONG Local buffer activity, for temporary tables.
temp_blks_read / temp_blks_written LONG Temporary file activity, for example from a large sort or hash.
neon_getpage_count LONG Number of page requests sent to the pageserver.
neon_file_cache_hits LONG Number of pages served from the local file cache instead of the pageserver.
neon_getpage_wait_us LONG Time spent waiting on pageserver page requests, in microseconds.
backend_pid LONG The backend process ID that ran this query.
lock_wait_count LONG Number of times this execution waited on a lock.
userid / dbid LONG The connecting role's and database's object IDs.
query_text STRING The query text. Masked. See Masked statement text.
plan_json STRING The full query plan, as JSON. Masked. See Masked statement text.
usename / datname STRING The connecting role's and database's names.
application_name / client_addr STRING The connecting application's name and client address.
wait_events STRING Wait events observed during this execution.
parameters STRING Bind parameters used in the query, if any. Masked. See Masked statement text.
blocking_pids STRING PIDs of sessions that blocked this execution, if any.
neon_branch_id / neon_endpoint_id / neon_timeline_id STRING Internal Neon identifiers for the branch, endpoint, and timeline.
cpu_user_time_ms / cpu_sys_time_ms DOUBLE CPU time consumed by this execution, split into user and system time.
voluntary_csw / involuntary_csw LONG Context switches during this execution.
trace_id / span_id / trace_flags / service_name STRING / STRING / LONG / STRING OpenTelemetry-style tracing identifiers, if the query was part of a traced request.
timeline_id LONG Internal timeline identifier.
truncated BOOLEAN Whether this row's data (for example, a large plan) was truncated before being written.

Query statistics

Aggregated per-query statistics, sourced from the standard pg_stat_statements extension.

pg_stat_statements_counters

Column Type Description
userid / dbid LONG The connecting role's and database's object IDs.
queryid LONG Identifies the query. Matches queryid in plan_history and active_session_history.
toplevel BOOLEAN Whether this row is for a top-level statement, as opposed to one nested inside a PL/pgSQL function or procedure. See the PL/pgSQL limitation.
query STRING The query text. Masked. See Masked statement text.
calls LONG Number of times this query was executed in the interval.
plans LONG Number of times this query was planned in the interval.
rows LONG Total rows returned or affected.
total_exec_time / total_plan_time DOUBLE Total time spent executing and planning this query in the interval.
shared_blks_hit / shared_blks_read / shared_blks_dirtied / shared_blks_written LONG Shared buffer cache activity.
local_blks_hit / local_blks_read / local_blks_dirtied / local_blks_written LONG Local buffer activity, for temporary tables.
temp_blks_read / temp_blks_written LONG Temporary file activity.
wal_bytes / wal_fpi / wal_records LONG Write-ahead log volume generated by this query.
jit_emission_count / jit_emission_time / jit_functions / jit_generation_time / jit_inlining_count / jit_inlining_time / jit_optimization_count / jit_optimization_time Mixed JIT (just-in-time) compilation statistics, if JIT was used for this query.

Schema changes

Schema changes: what changed, when, and by whom. This is what Genie reads to trace an incident back to a schema change, whether from a deploy or a manual edit.

ddl_history

Column Type Description
row_position LONG Ordinal position of this row within its batch.
command_tag STRING The type of DDL command, for example ALTER TABLE.
object_type / object_name STRING The kind and name of the object that changed.
schema_name STRING The Postgres schema the object belongs to.
query_text STRING The full DDL statement. Masked. See Masked statement text.
duration_ms DOUBLE How long the DDL statement took to run.
userid / usename LONG / STRING The connecting role's object ID and name.
application_name STRING The connecting application's name.
backend_pid LONG The backend process ID that ran the statement.
dbid LONG The database's object ID.
search_path STRING The session's search path at the time.
ddl_json STRING A structured, parsed representation of the DDL statement. Masked. See Masked statement text.

Note

DDL capture has a real gap: a change made right around a compute resuming from scale-to-zero can be missed. If a schema change doesn't appear here even though you know it happened, that's the most likely reason. See Limitations.

Postgres logs

Raw Postgres log messages, including the specific error behind a failure.

postgres_logs

Column Type Description
sample_seq LONG Ordinal position of this row within its batch.
elevel LONG Internal log level encoding.
severity STRING The log severity, for example ERROR or LOG.
sqlstate STRING The Postgres SQLSTATE error code, for example 42703 for an undefined column.
backend_pid / leader_backend_pid LONG The backend process ID, and its parallel query leader's PID if applicable.
userid / usename LONG / STRING The connecting role's object ID and name.
datid / datname LONG / STRING The database's object ID and name.
application_name / client_addr STRING The connecting application's name and client address.
schema_name / table_name / column_name / constraint_name / datatype_name STRING The schema, table, column, constraint, or data type referenced by the error, when Postgres has one.
funcname STRING The function the error occurred in, if applicable.
filename / lineno STRING / LONG The Postgres source file and line the error originated from.
message STRING The log message text.
detail / hint / context STRING Additional detail, a suggested fix, and the error's context, when Postgres has them.
internalquery STRING An internally generated query related to the error, if applicable.
statement STRING The statement that triggered this log entry. Masked. See Masked statement text.
truncated_mask LONG Indicates which fields, if any, were truncated before being written.

Note

The structured schema_name, table_name, and column_name fields aren't populated for every error type. A plain "column does not exist" parse error (SQLSTATE 42703), for example, leaves them empty even though message names the column directly. Don't rely on these fields being populated. Check message first.

Compute metrics

Compute-level resource usage: CPU, memory, disk and network I/O, and the local file cache (LFC), Lakebase's compute-side cache in front of the pageserver.

compute_counters

Column Type Description
backpressure_throttling_seconds DOUBLE Time the storage layer spent throttling writes in this interval. A non-zero value means write volume exceeded what storage can absorb.
host_busiest_cpu_seconds / host_cpu_seconds DOUBLE CPU time consumed, for the busiest core and in total.
host_disk_read_bytes / host_disk_written_bytes DOUBLE Disk I/O volume.
host_network_receive_bytes / host_network_transmit_bytes DOUBLE Network I/O volume.
lfc_evictions / lfc_hits / lfc_misses / lfc_writes LONG Local file cache activity for this interval.
replica_lfc_redo_evictions LONG Local file cache evictions caused by replica redo, if this compute is a read replica.

compute_gauges

Column Type Description
current_lsn LONG The current write-ahead log position.
replica_wal_replay_lsn / replica_wal_receive_lsn LONG WAL replay and receive positions, if this compute is a read replica.
max_connections LONG The configured connection limit.
lfc_working_set_size_5m / lfc_working_set_size_15m / lfc_working_set_size_60m LONG Estimated working set size over the last 5, 15, and 60 minutes.
lfc_size_limit / lfc_allocated / lfc_used LONG Local file cache capacity and current usage.
total_size_limit / total_size LONG Overall storage capacity and current usage.
replication_lag_bytes / replication_lag_time_ms LONG / DOUBLE Replication lag, if this compute is a read replica.
host_logical_cpus / host_memory_total_bytes DOUBLE The compute's provisioned CPU and memory.

Database metrics

Per-database activity: rows changed, transactions, sessions, and vacuum health. This is where table bloat and autovacuum signals live.

database_counters

Column Type Description
datid / datname LONG / STRING The database's object ID and name.
active_time_ms / idle_in_transaction_time_ms / session_time_ms DOUBLE Time spent active, idle in an open transaction, and connected overall.
deadlocks LONG Number of deadlocks detected.
rows_deleted / rows_fetched / rows_inserted / rows_returned / rows_updated LONG Row-level activity for the interval.
sessions_abandoned / sessions_established / sessions_fatal / sessions_killed LONG Session lifecycle counts.
shared_blks_hit / shared_blks_read LONG Shared buffer cache activity.
xact_commit / xact_rollback LONG Committed and rolled-back transactions.

database_gauges

Column Type Description
datid / datname LONG / STRING The database's object ID and name.
numbackends LONG Current number of connections to this database.
size LONG The database's on-disk size.
oldest_frozen_xid_age / oldest_mxid_age LONG Transaction ID and multixact ID age of the oldest unfrozen row. Rising values are the earliest signal of vacuum falling behind, well before bloat becomes visible in table size.

Limitations

Note

Passwords are redacted as a statement is captured, so a captured CREATE ROLE reads PASSWORD '***'. Redaction depends on detecting the password in the statement, and a malformed statement can escape detection.

Azure Databricks recommends OAuth roles, which authenticate Azure Databricks identities with tokens and leave no password to type.

The following limitations apply:

  • COPY operations are not captured in query plan history.

  • A query executed inside a PL/pgSQL function or procedure isn't captured in query-level telemetry unless it also runs as a top-level statement. Postgres's own statement tracking only counts top-level statements by default.

  • Branches and endpoints appear in telemetry and insights by internal ID (for example, br-odd-mud-y2icrn1h), not by a display name like "production." Cross-reference the ID in the Lakebase UI if you must confirm which branch or endpoint an insight or table row refers to.

  • GUCs that enable telemetry capture might not propagate automatically to an existing compute. If telemetry doesn't appear after the preview is enabled, connect to the project and check:

    SHOW neon_monitor.log_capture_enabled;
SHOW databricks.o11y_exporter_enabled;

    Both must return on. If either returns off, restart the compute endpoint.

Legacy observability configurations

Before telemetry moved to system tables, you could create an observability configuration that captured telemetry to a catalog and schema in your own Unity Catalog and provided prebuilt dashboards. Existing configurations keep working, and their dashboards remain available. You can no longer create new observability configurations, and capture to system tables doesn't create dashboards.

Next steps

  • Last updated on