Structured Streaming concepts

Apache Spark Structured Streaming lets you build streaming applications and pipelines with the same Spark APIs you use for batch processing. On Azure Databricks, it supports two processing modes:

• Real-time mode (RTM): Designed for sub-second operational workloads, such as fraud detection and real-time personalization. RTM processes records as they arrive, with end-to-end latency as low as five milliseconds. See Real-time mode concepts.
• Micro-batch mode: Designed for analytical pipelines and extract, transform, and load (ETL) workloads. Micro-batch mode processes streams in small batches, balancing compute cost with latency requirements measured in seconds or minutes. Micro-batch processing is the default mode.

Both modes use the same Structured Streaming APIs. For queries supported by both modes, you can switch modes through the trigger and compute configuration. See Set up real-time mode.

Structured Streaming also powers streaming in Lakeflow pipelines, built on Apache Spark™ Declarative Pipelines. Pipelines manage orchestration, checkpoints, and recovery, and you define transformations in SQL or Python.

For a step-by-step tutorial, see Run your first Structured Streaming workload.

Read from a data stream

Use Structured Streaming to incrementally ingest data from supported data sources.

Feature	Description
Auto Loader	Incrementally and efficiently process new data files as they arrive in cloud storage.
Delta Lake table streaming reads and writes	Use Delta Lake tables as streaming sources and sinks with exactly-once processing guarantees.
SQL and Spark APIs	Connect to message buses, queues, and enterprise applications using SQL and Spark APIs.
Micro-batch size	Limit input rates to maintain consistent batch sizes and prevent processing delays.

Write to a data sink

Configure how Structured Streaming delivers data to target systems.

Feature	Description
Checkpoints	Store processing state to enable fault tolerance and exactly-once delivery semantics.
Output mode	Choose between append, update, and complete modes for stateful streaming queries.
Trigger intervals	Set trigger intervals to balance latency and cost for your processing requirements.
Real-time mode in Structured Streaming	Process data for real-time workloads with end-to-end latency as low as five milliseconds.

Stateful and stateless processing

Stateless queries process rows without retaining state. Stateful queries maintain intermediate state for aggregations, joins, and deduplication.

Feature	Description
Stateless streaming queries	Optimize queries that process data without maintaining intermediate state.
Watermarks	Control how long Structured Streaming waits for late-arriving data in stateful operations.
Stateful streaming	Manage aggregations, stream-stream joins, and deduplication using stateful operators.

Monitor and manage

Track query performance, apply optimizations, and govern data access for production Structured Streaming workloads.

Feature	Description
Monitor with StreamingQueryListener	Track query progress and performance metrics using the Spark UI and listener API.
Govern with Unity Catalog	Configure Unity Catalog for streaming workloads with governance and access control.