Real-time mode in Structured Streaming

Real-time mode is a Structured Streaming trigger type with end-to-end latency as low as five milliseconds. Use it for operational workloads that must respond to streaming data immediately, such as fraud detection and real-time personalization.

Real-time mode uses the same Structured Streaming APIs as micro-batch processing. You enable it by setting the real-time trigger on a streaming query. To compare it with the other Structured Streaming trigger types, see Configure Structured Streaming trigger intervals.

Real-time mode is also available in Lakeflow pipelines. See Use real-time mode in Lakeflow pipelines.

Get started

Page	Description
Set up real-time mode	Configure classic compute, enable the real-time trigger, and size your cluster.
Real-time mode concepts	Learn how real-time mode works and when to use it instead of micro-batch mode.
Tutorial: Run a real-time streaming workload	Run your first streaming query with the real-time trigger and a rate source.

Build real-time applications

Page	Description
Optimize and monitor real-time mode query performance	Tune compute, reduce end-to-end latency, and monitor query performance with built-in metrics.
Real-time mode limitations	Understand known limitations for sources, unions, and mapPartitions.
Real-time mode reference	Review supported environments, languages, compute types, sources, sinks, and operators.
Real-time mode examples	Explore code examples for Kafka, Kinesis, Lakebase, stateful queries, and custom sinks.

Additional resources

• Structured Streaming concepts
• Configure Structured Streaming trigger intervals
• Use real-time mode in Lakeflow pipelines