Schema management

This deep dive explains how Zerobus Ingest in Lakeflow Connect validates incoming records against your Delta table schema, and how to design your schema for partial or evolving data.

The table is the contract

Your Delta table schema is the authoritative contract for what Zerobus Ingest accepts. Zerobus Ingest gates every record against that contract, but you choose how strict or how accepting the contract is. The same service can enforce a rigid schema, accept a flexible subset of columns, or catch everything that doesn't fit, depending on how you define your table.

• Zerobus Ingest gates the data. It validates every record against the target table and rejects anything that doesn't fit. It never guesses or drops columns silently.
• You define the contract. Marking columns required or nullable, and adding a rescue column, is how you decide what "fits."
• Zerobus Ingest never augments your table. It does not add columns, change types, or evolve the schema to accommodate a record. You evolve what Zerobus Ingest accepts by evolving the table, not the other way around.

The next section shows three ways to shape that contract, from most accepting to strict to catch-all.

How records are matched to the table

A record must fit into the destination table: it must contain, at minimum, all of the non-nullable columns in the table. Columns that are nullable in the table can be omitted from the record, and are written as NULL. Omitting a nullable column is treated as a non-breaking change, so you can add nullable columns to a table and keep ingesting older records that don't include them.

Zerobus Ingest returns an error when a record does not fit the table. This includes:

• A missing non-nullable column.
• A column name that does not exist in the Delta table (unless you configure a rescue column).
• A column whose type is not compatible with the Delta table. For the supported Delta and Protobuf data types, see Supported data types.

Three ways to shape the contract

How you define the table determines how strict or accepting ingestion is. The three scenarios below go from most accepting to strict to catch-all.

Scenario 1: All columns optional (accept a subset)

Make every column nullable. Producers can then send any subset of the columns, and omitted columns are written as NULL. Records that include a column the table doesn't have are still rejected.

CREATE TABLE main.default.air_quality (
  device_name STRING,
  temp INT,
  humidity INT);

• {"device_name": "sensor-1", "temp": 22, "humidity": 55}: accepted. All columns present.
• {"device_name": "sensor-1"}: accepted. temp and humidity are nullable, so they're written as NULL.
• {"device_name": "sensor-1", "temp": 22, "region": "us-west"}: rejected. region doesn't exist in the table.

Scenario 2: Required columns (enforce specific fields)

Mark columns NOT NULL to require them. Every record must supply those columns, or it's rejected. This is the strict end of the spectrum: use it when a field must always be present.

CREATE TABLE main.default.air_quality (
  device_name STRING NOT NULL,
  temp INT NOT NULL,
  humidity INT);

• {"device_name": "sensor-1", "temp": 22, "humidity": 55}: accepted. All required columns present.
• {"device_name": "sensor-1", "temp": 22}: accepted. humidity is nullable, so it's written as NULL.
• {"device_name": "sensor-1"}: rejected. temp is non-nullable and is missing.

Scenario 3: Rescue column (catch everything else)

Add a VARIANT rescue column to capture fields that don't fit the schema, instead of rejecting the record. Fields that match the table are written to their columns as usual. Any extra or non-conforming fields are grouped into the rescue column as a JSON object. This is the most accepting end of the spectrum: extra fields and nullable-column type mismatches are captured instead of being rejected. A record is still rejected if it omits a required (non-nullable) column, because the rescue column can't supply a value the schema requires. The rescue column is in Beta and currently supports JSON-format ingestion.

• {"device_name": "sensor-1", "temp": 22, "region": "us-west"}: accepted. region doesn't exist in the table, so it's captured in the rescue column instead of being rejected.

For how to configure a rescue column and the exact rules, see Zerobus rescue column.

Schema evolution

Zerobus Ingest doesn't auto-evolve your target table. When your data shape changes, evolve the table first (for example, with ALTER TABLE), then send records against the new schema.

Adding a nullable column is a non-breaking change: existing producers that don't send the new column keep working, and their records get NULL for it. This lets you roll out schema changes and producer changes independently.

Protobuf schema

When you ingest with Protocol Buffers (protobuf), the same fit rule applies to your protobuf message definition: it must contain at minimum all of the non-nullable columns in the Delta table, and it may omit nullable ones.

The following also apply to the protobuf schema:

• Zerobus Ingest does not support proto schemas with more than 2000 columns.
• Zerobus Ingest only supports table and column names with ASCII letters, digits, and underscores.
• Zerobus Ingest does not support using a different proto schema for "stream creation" and "ingest record" operations.