Capture nonconformant fields with the Zerobus rescue column

Important

The rescue column is in Beta. It is currently supported only for ingestion in JSON format.

By default, Zerobus Ingest rejects any record that contains fields that don't match the target table's schema. The rescue column feature lets you capture those non-conforming fields instead of losing them. Fields that don't fit the schema are grouped into a JSON object and stored in a designated rescue column as a VARIANT type.

Configure a rescue column

To designate a column as the rescue column, the column must:

• Allow null values.
• Use the VARIANT type.
• Have the zerobus-rescue tag applied in Unity Catalog.

Exactly one column should meet all three criteria. Zerobus handles the other cases as follows:

• No column meets all three criteria: The rescue column feature is inactive, and Zerobus rejects non-conforming fields as it would without the feature.
• Multiple columns meet all three criteria: Zerobus selects one arbitrarily. To avoid ambiguity, verify that only one column qualifies.

Changes to column tagging might take up to 5 minutes to take effect.

When Zerobus writes to the rescue column

Zerobus writes a field to the rescue column when either of the following occurs:

• The record contains a field that isn't present in the table schema.
• The record contains a field that's present in the table schema, but the value's type doesn't match and the target column is nullable.

Fields that conform to the schema are written to their respective columns normally. Only non-conforming fields go to the rescue column.

Example: type mismatch and extra field

The following example shows a type mismatch and an extra field, both routed to the rescue column. Consider a table with the following schema, where rescue is nullable, uses the VARIANT type, and has the zerobus-rescue tag applied in Unity Catalog:

CREATE TABLE main.default.air_quality (
  device_name STRING NOT NULL,
  temp INT,
  humidity LONG,
  rescue VARIANT
);

This example ingests the following JSON record:

{
  "device_name": "sensor-1",
  "temp": "72F",
  "humidity": 87,
  "extra_field": "some value"
}

The record produces the following row in the target Delta table:

device_name	temp	humidity	rescue
sensor-1	null	87	{"extra_field": "some value", "temp": "72F"}

Each column's outcome:

• device_name is written normally because the payload value matches the column's STRING type.
• temp is written as null because the payload value ("72F", a string) doesn't match the column's INT type.
• humidity is written normally because the payload value matches the column's type.
• extra_field isn't in the table schema, so it's written to rescue instead of being rejected.

Zerobus groups both non-conforming fields, temp and extra_field, into the rescue column's JSON object.

Limitations

The following limitations apply to the rescue column feature:

• The rescue column is reserved, meaning that the record payload can't provide an explicit non-null value for it. If a record does, it's rejected with an error. A null value for the rescue column is allowed and is ignored, the same as any absent or null field.
• A field set to null in the JSON payload is treated the same as an absent field and is not written to the rescue column.
• If the payload contains multiple entries for the same key, the last value wins, following the standard JSON parsing convention.