readBetweenOffsets (SimpleDataSourceStreamReader)

Reads all available data between a specific start offset and end offset.

This method is invoked during failure recovery to re-read a batch deterministically.

Added in Databricks Runtime 15.3

Syntax

readBetweenOffsets(start: dict, end: dict)

Parameters

Parameter	Type	Description
start	dict	The start offset to read from.
end	dict	The offset at which reading stops.

Returns

Iterator[Tuple]

All records between the start offset and end offset.