OpenTelemetry table reference for Zerobus Ingest

This page provides reference information for the OpenTelemetry (OTLP) table schemas and data mapping used by Zerobus Ingest OTLP.

Table schema

When OTLP data arrives, Zerobus Ingest converts each record from the nested OTLP resource/scope/record hierarchy into a flat, denormalized row. Resource attributes and instrumentation scope information are embedded directly in each row, making the data immediately queryable without joins.

All attribute fields (attributes, resource.attributes, instrumentation_scope.attributes, body for logs, metadata for metrics) are stored as VARIANT columns. VARIANT is a semi-structured type in Delta Lake that stores JSON data while preserving the original types. For how each OTLP value type is stored, see Attribute values.

Each record is augmented with Databricks-specific fields:

Field Description Source
record_id A system-generated ID for unique identification and time-ordered sorting. Generated based on time
time Timestamp in microseconds from the Unix epoch. Timestamp (in microseconds) derived from start_time_unix_nano (spans) or time_unix_nano (logs, metrics). For a log record without time_unix_nano, observed_time_unix_nano is used.
date Date partition column, for efficient time-range filtering. Derived from time
service_name Top-level column for efficient filtering by service name, as defined in the OTel semantic convention. Extracted from resource.attributes["service.name"]

Numeric precision and signed types

Delta types differ from OTLP types in the following ways:

  • Integer metric values lose precision above 2^53. Gauge, Sum, and Exemplar data points that report their value as as_int, a 64-bit integer, are converted to DOUBLE, since Zerobus Ingest stores all metric values in a single floating-point column. DOUBLE only represents integers exactly up to 2^53, about 9 quadrillion, so larger as_int values lose exact precision after conversion. This mainly matters for very large cumulative counters, such as lifetime byte or request totals.
  • Unsigned OTLP fields are stored as signed. INT and BIGINT are always signed in Delta, but OTLP defines some numeric fields as unsigned (uint32, uint64, fixed64) or as very large fixed-width counters. Fields such as *_time_unix_nano, flags, dropped_attributes_count, etc., are stored as-is in a signed INT or BIGINT column. A value above the signed maximum (i32::MAX for INT, i64::MAX for BIGINT) wraps to a negative number instead of erroring.

Schema mapping

Zerobus Ingest maps OTLP data to Delta table columns as described below.

Denormalization

In the OTLP protocol, telemetry data is nested like so.

ResourceSpans (or ResourceLogs, ResourceMetrics)
  └── Resource (attributes, schema_url)
       └── ScopeSpans (or ScopeLogs, ScopeMetrics)
            └── InstrumentationScope (name, version, attributes)
                 └── Span (or LogRecord, Metric)

Zerobus Ingest flattens this hierarchy so that each row contains the full context:

  • resource: A struct containing the resource attributes (as VARIANT) and dropped_attributes_count.
  • resource_schema_url: The schema URL from the enclosing ResourceSpans, ResourceLogs, or ResourceMetrics.
  • instrumentation_scope: A struct containing the scope name, version, attributes (as VARIANT), and dropped_attributes_count.
  • span_schema_url / log_schema_url / metric_schema_url: The schema URL from the enclosing ScopeSpans, ScopeLogs, or ScopeMetrics.

Metric data points

Each metric data point is stored as its own row. The metric's name, description, unit, and metadata are repeated on every row, and the data point's start_time_unix_nano and time_unix_nano are stored in the top-level columns of the same name.

metric_type is gauge, sum, histogram, exponential_histogram, or summary. The data point is stored in the struct column with that name, and the other four struct columns are NULL.

Attribute values

Attribute values and log bodies are OTLP AnyValue values, which are stored as follows:

OTLP value type Stored as
string_value String
bool_value Boolean
int_value Integer
double_value Double. NaN, positive infinity, and negative infinity are stored as the strings "NaN", "Infinity", and "-Infinity".
array_value Array
kvlist_value Object
bytes_value Lowercase hex-encoded string. For example, the bytes 0x01 0xFF are stored as "01ff".

A bytes_value can't be told apart from a string_value that holds the same text. To get the original bytes, decode it with unhex, for example unhex(attributes:['payload']::string).

If the same key appears more than one time in an attribute list or a kvlist_value, the last value is stored.

ID encoding

trace_id, span_id, and parent_span_id are stored as lowercase hex-encoded strings:

  • trace_id: 32-character hex string (16 bytes)
  • span_id and parent_span_id: 16-character hex string (8 bytes)

An ID is invalid if it has the wrong length or is all zeros. For a span's own trace_id and span_id, see Rejected records. Other missing and invalid IDs are handled as follows:

  • A missing or invalid parent_span_id is stored as NULL, so the span is stored as a root span.
  • A missing or invalid trace_id or span_id on a log record is stored as NULL, and the log record is ingested.
  • The trace_id and span_id of span links and exemplars are stored hex-encoded as received, without validation.

Enum encoding

kind, status.code, and severity_number are stored as their string names as defined in the OTLP specification, for example SPAN_KIND_SERVER, STATUS_CODE_OK, and SEVERITY_NUMBER_INFO. A value that the OTLP specification doesn't define is stored as its number, for example 42.

aggregation_temporality is stored without the AGGREGATION_TEMPORALITY_ prefix, as DELTA or CUMULATIVE.

Rejected records

Zerobus Ingest rejects a record that fails any of the following checks. For how rejected records are reported, see Partial success.

A span is rejected when:

  • trace_id or span_id is missing or invalid.
  • start_time_unix_nano is missing.

A metric data point is rejected when:

  • Its metric has no name, or has an aggregation_temporality value that the OTLP specification doesn't define. All of the metric's data points are rejected.
  • time_unix_nano is missing or 0.
  • A gauge or sum data point has no value.
  • A histogram data point has explicit_bounds but no bucket_counts, has a number of bucket_counts other than the number of explicit_bounds plus one, or has bucket_counts that don't add up to count.
  • An exponential histogram data point has positive bucket counts, negative bucket counts, and zero_count that don't add up to count.
  • A histogram, exponential histogram, or summary data point has a count of 0 and a sum other than 0.

Data points with the NO_RECORDED_VALUE flag set skip the value, count, sum, and bucket checks.

Log records aren't rejected for their content. For how a log record's missing or invalid IDs are stored, see ID encoding.

Next steps