Zerobus rescue column

Important

The rescue column is in Beta and currently supports ingestion in JSON format. Workspace admins can enable it from the Previews page by opting into the Zerobus rescue column preview. See Manage Azure Databricks previews.

By default, Zerobus Ingest in Lakeflow Connect rejects any record that contains fields that don't match the target table's schema. The rescue column feature lets you capture those non-conforming fields instead of losing them. Fields that don't fit the schema are grouped into a JSON object and stored in a designated rescue column as a VARIANT type.

The rescue column currently supports JSON-format ingestion. If you ingest with Protocol Buffers (protobuf) or Apache Arrow, plan your schema so records fit the table, because non-conforming fields are rejected rather than rescued.

The rescue column is one of three ways to shape how strictly your table validates incoming data. For the full picture, see Three ways to shape the contract.

Configure a rescue column

To designate a column as the rescue column, the column must:

  • Allow null values.
  • Use the VARIANT type.
  • Have the zerobus-rescue tag applied in Unity Catalog.

Exactly one column should meet all three criteria. Zerobus handles the other cases as follows:

  • No column meets all three criteria: The rescue column feature is inactive, and Zerobus rejects non-conforming fields as it would without the feature.
  • Multiple columns meet all three criteria: Zerobus selects one arbitrarily. To avoid ambiguity, verify that only one column qualifies.

Changes to column tagging might take up to 5 minutes to take effect.

When Zerobus writes to the rescue column

Zerobus Ingest routes each field in a record based on how it fits the table schema:

Field in the record Outcome
Matches a column in the table (name and type) Written to that column normally.
Not present in the table schema Captured in the rescue column.
Present in the table schema, but the value's type doesn't match, and the target column is nullable Captured in the rescue column.

Only non-conforming fields go to the rescue column. Fields that conform to the schema are always written to their own columns.

Example: type mismatch and extra field

The following example shows a type mismatch and an extra field, both routed to the rescue column. Consider a table with the following schema, where rescue is nullable, uses the VARIANT type, and has the zerobus-rescue tag applied in Unity Catalog:

CREATE TABLE main.default.air_quality (
  device_name STRING NOT NULL,
  temp INT,
  humidity LONG,
  rescue VARIANT
);

This example ingests the following JSON record:

{
  "device_name": "sensor-1",
  "temp": "72F",
  "humidity": 87,
  "extra_field": "some value"
}

The record produces the following row in the target Delta table:

device_name temp humidity rescue
sensor-1 null 87 {"extra_field": "some value", "temp": "72F"}

Each column's outcome:

  • device_name is written normally because the payload value matches the column's STRING type.
  • temp is written as null because the payload value ("72F", a string) doesn't match the column's INT type.
  • humidity is written normally because the payload value matches the column's type.
  • extra_field isn't in the table schema, so it's written to rescue instead of being rejected.

Zerobus groups both non-conforming fields, temp and extra_field, into the rescue column's JSON object.

Limitations

The following limitations apply when using the rescue column:

  • The rescue column is reserved, meaning that the record payload can't provide an explicit non-null value for it. If a record does, it's rejected with an error. A null value for the rescue column is allowed and is ignored, the same as any absent or null field.
  • A field set to null in the JSON payload is treated the same as an absent field and is not written to the rescue column.
  • If the payload contains multiple entries for the same key, the last value wins, following the standard JSON parsing convention.
  • Last updated on