Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Supported languages
Real-time mode supports Scala, Java, and Python.
Compute types
Real-time mode supports the following compute types:
| Compute type | Supported |
|---|---|
| Dedicated (formerly: single user) | ✓ |
| Standard (formerly: shared) | ✓ (only Python) |
| Lakeflow Spark Declarative Pipelines Classic | Not supported as Structured Streaming. Supported through pipeline configuration. See Use real-time mode in Lakeflow Spark Declarative Pipelines. |
| Lakeflow Spark Declarative Pipelines Serverless | Not supported as Structured Streaming. Supported through pipeline configuration. See Use real-time mode in Lakeflow Spark Declarative Pipelines. |
| Serverless | Not supported |
For latency-sensitive workloads with UDFs, Databricks recommends that you use dedicated access mode. See Table functions.
Execution modes
Real-time mode supports update mode only:
| Execution mode | Supported |
|---|---|
| Update mode | ✓ |
| Append mode | Not supported |
| Complete mode | Not supported |
Sources and sinks
Real-time mode supports the following sources and sinks:
| Source or sink | As source | As sink |
|---|---|---|
| Apache Kafka | ✓ | ✓ |
| Event Hubs (using Kafka connector) | ✓ | ✓ |
| Kinesis | ✓ (only EFO mode) | Not supported |
| AWS MSK | ✓ | Not supported |
| Delta | Not supported | Not supported |
| Google Pub/Sub | Not supported | Not supported |
| Apache Pulsar | Not supported | Not supported |
Arbitrary sinks (using forEachWriter) |
Not applicable | ✓ |
Operators
Real-time mode supports most Structured Streaming operators:
Stateless operations
| Operator | Supported |
|---|---|
| Selection | ✓ |
| Projection | ✓ |
mapPartitions |
Not supported (see limitation) |
| Union | ✓ (with some limitations) |
UDFs
| Operator | Supported |
|---|---|
| Scala UDF | ✓ (with some limitations) |
| Python UDF | ✓ (with some limitations) |
Aggregation
| Function | Supported |
|---|---|
| sum | ✓ |
| count | ✓ |
| max | ✓ |
| min | ✓ |
| avg | ✓ |
| Aggregation functions | ✓ |
Windowing
| Operator | Supported |
|---|---|
| Tumbling | ✓ |
| Sliding | ✓ |
| Session | Not supported |
Deduplication
| Operator | Supported |
|---|---|
| dropDuplicates | ✓ |
| dropDuplicatesWithinWatermark | ✓ |
Stream to table join
| Operator | Supported |
|---|---|
| Inner join | ✓ |
| Outer Join | ✓ |
| Broadcast table join (table size of 10mb or less) | ✓ |
| Table join (without broadcast) | Not supported |
Stream to stream join
| Operator | Supported |
|---|---|
| Inner join | ✓ (Databricks Runtime 18 and above, with some configurations) |
| Outer join | Not supported |
Note
To use stream to stream joins in real-time mode, you must set additional Spark configurations. For more information on the configurations and for the requirements to run multiple streams, see Stream to stream joins.
Arbitrary stateful operator
| Operator | Supported |
|---|---|
| (flat)MapGroupsWithState | Not supported |
| transformWithState | ✓ (with some differences) |
User defined sinks
| Sink | Supported |
|---|---|
| forEach | ✓ |
| forEachBatch | Not supported |
Special considerations
Some operators and features have specific considerations or differences when used in real-time mode.
transformWithState in real-time mode
For building custom stateful applications, Databricks supports transformWithState, an API in Apache Spark Structured Streaming. See Build a custom stateful application for more information about the API and code snippets.
However, the API behaves differently in real-time mode than in micro-batch queries.
- Real-time mode calls the
handleInputRows(key: String, inputRows: Iterator[T], timerValues: TimerValues)method for each row.- The
inputRowsiterator returns a single value. Micro-batch mode calls it once for each key, and theinputRowsiterator returns all values for a key in the micro batch. - Account for this difference when writing your code
- The
- Event time timers are not supported in real-time mode.
transformWithStateInPandasis not supported in real-time mode. Use the row-basedtransformWithStateAPI instead, which usesRowobjects rather than pandas DataFrames.- In real-time mode, timers are delayed in firing depending on data arrival:
- If a timer is scheduled for 10:00:00 but no data arrives, the timer doesn't fire immediately.
- If data arrives at 10:00:10, the timer fires with a 10-second delay.
- If no data arrives and the long-running batch is terminating, the timer fires before the batch terminates.
Note
In Databricks Runtime 18.1 and below, if you use transformWithState and real-time mode for Python with low throughput, less than 5 records per second, you might see increased latencies of up to a few hundred milliseconds. Databricks recommends upgrading to Databricks Runtime 18.2 and above to resolve.
Python UDFs in real-time mode
Databricks supports the majority of Python user-defined functions (UDFs) in real-time mode:
Stateless
| UDF type | Supported |
|---|---|
| Python scalar UDF (User-defined scalar functions - Python) | ✓ |
| Arrow scalar UDF | ✓ |
| Pandas scalar UDF (pandas user-defined functions) | ✓ |
Arrow function (mapInArrow) |
✓ |
| Pandas function (Map) | ✓ |
Stateful grouping (UDAF)
| UDF type | Supported |
|---|---|
transformWithState (only Row interface) |
✓ |
transformWithStateInPandas |
Not supported. Use the row-based transformWithState API instead, which uses Row objects rather than pandas DataFrames. See transformWithStateInPandas not supported for details. |
applyInPandasWithState |
Not supported |
Non-stateful grouping (UDAF)
| UDF type | Supported |
|---|---|
apply |
Not supported |
applyInArrow |
Not supported |
applyInPandas |
Not supported |
Table functions
| UDF type | Supported |
|---|---|
| UDTF (Python user-defined table functions (UDTFs)) | Not supported |
| UC UDF | Not supported |
There are several points to consider when using Python UDFs in real-time mode:
- To minimize latency, set the Arrow batch size (
spark.sql.execution.arrow.maxRecordsPerBatch) to 1.- Trade-off: This configuration optimizes for latency at the expense of throughput. For most workloads, this setting is recommended.
- Increase the batch size only if a higher throughput is required to accommodate input volume, accepting the potential increase in latency.
- Pandas UDFs and functions do not perform well with an Arrow batch size of 1.
- If you use pandas UDFs or functions, set the Arrow batch size to a higher value (for example, 100 or higher).
- This implies higher latency. Databricks recommends using an Arrow UDF or function if possible.
transformWithStateInPandasis not supported in real-time mode. Use the row-basedtransformWithStateAPI instead, which usesRowobjects rather than pandas DataFrames. SeetransformWithStateInPandasnot supported and Real-time mode examples for a working Python example using the row-based API.- For latency-sensitive workloads with UDFs, Databricks recommends that you use dedicated access mode. In standard access mode, security isolation overhead might slow UDF performance.