Lakehouse Real-Time

Important

This feature is in Beta. Reach out to your Databricks account team to enable this feature in your account.

Lakehouse//RT is in active development. Performance characteristics and the supported feature set will change before general availability.

Lakehouse Real-Time (Lakehouse//RT) is a serverless compute built for low-latency, high-concurrency use cases, such as serving analytical data to custom applications, running operational analytics, or powering BI dashboards that require sub-second responses for hundreds to thousands of concurrent users.

Lakehouse//RT offers sub-second latency on SQL read queries against your Unity Catalog tables that use Delta Lake or Apache Iceberg formats in cloud storage. You create and manage Lakehouse//RT much like you do other SQL warehouses. A workspace admin or privileged user creates one or more per workspace and assigns permissions to users.

Requirements

To use Lakehouse//RT, you must:

  • Be in a supported region.
  • Have the Lakehouse//RT Beta enabled in your workspace.

Enable Lakehouse//RT in your workspace

Workspace admins can enable the Lakehouse//RT Beta in your workspace:

  1. In your workspace menu (top-right corner), go to Previews.
  2. Search for Lakehouse RT.
  3. Enable the preview.

After you enable the preview, the Real-Time warehouse type becomes available in the SQL warehouse creation flow for your workspace.

Create a Lakehouse//RT warehouse

To create a Lakehouse//RT warehouse:

  1. Go to Compute > SQL Warehouses > Create SQL Warehouse.
  2. Select Real-Time.
  3. Select a size: Small, Medium, Large, or X-Large, depending on the performance your queries require.
  4. Enter a name for the warehouse.
  5. Click Create.

To assign permissions, grant Can use, Can monitor, or Can manage to users and groups, the same as a SQL warehouse.

Note

You cannot currently upgrade an existing SQL warehouse to Lakehouse//RT or downgrade an existing Lakehouse//RT warehouse to another warehouse type.

Monitor Lakehouse//RT activity

You can monitor Lakehouse//RT queries the same as any query run on a SQL warehouse:

  • Query history: Lakehouse//RT queries appear in the query history UI and the query history system table.
  • Query profiles: Open a Lakehouse//RT query in the query history UI to view its query profile.
  • Monitoring page: Monitor query throughput, queued queries, and query history on the monitoring page for each Lakehouse//RT warehouse.
  • Billing: Lakehouse//RT usage appears in the billing system tables with a sku_name of Lakehouse_Serverless.

Best practices

To get the best results from Lakehouse//RT, prepare your workloads before you move them over:

  • Validate on serverless SQL first. Run your queries on a serverless SQL warehouse and confirm that they run in a few seconds.
  • Use Unity Catalog managed tables. Managed tables with predictive optimization and liquid clustering ensure that your data is well clustered for your workload patterns.
  • Verify queries are selective. For sub-second latency, verify that your queries scan lesser amounts of data. Filter early with WHERE clauses, select only the columns you require, and lean on aggregations. Joining across tables is supported, but if you find your query getting complex or slow, consider using materialized views that pre-aggregate your data for faster latencies.
  • Check SQL coverage. Lakehouse//RT supports ANSI-compliant read queries only. Confirm that your workloads are ANSI compliant and avoid the unsupported statements, functions, and data types listed under Limitations.

Supported features

Tools and interfaces

You can select Lakehouse//RT from the compute picker in any of the following Azure Databricks features:

  • SQL editor
  • SQL notebooks
  • AI/BI Dashboards
  • Catalog Explorer
  • Alerts

Table types

Lakehouse//RT queries Unity Catalog data only. For best performance, use Unity Catalog managed tables, which provide the engine with the data layout it needs for low latency.

Lakehouse//RT supports the following table types:

  • Managed tables (Delta Lake and Apache Iceberg tables)
  • Materialized views and streaming tables
  • Metric views

Connectivity

Lakehouse//RT only accepts connections that use the Statement Execution API. It does not support the legacy Thrift protocol, so a driver that connects without explicitly using the Statement Execution API receives a 501 error.

You can connect to a Lakehouse//RT warehouse in the following ways:

Pricing

For pricing information, see the Lakehouse Real-Time pricing page.

Limitations

When a query uses an unsupported feature, Lakehouse//RT returns an error naming the feature. To successfully run the query, use a serverless SQL warehouse instead.

Tools and features

Lakehouse//RT does not support the following features:

  • Genie
  • Genie Spaces
  • Jobs tasks

Table types

The following table types are not supported during the Beta:

  • System tables
  • Delta Sharing tables
  • Tables in Unity Catalog default storage
  • External tables in Unity Catalog

Lakehouse//RT does not support the following table types:

  • Hive metastore tables (managed or external)
  • Foreign tables and query federation (Lakehouse Federation)
  • Temporary tables
  • Tables that use other data formats (CSV, JSON, Avro, Parquet, ORC, and text)

Drivers and connectors

Lakehouse//RT does not support the following drivers and connectors:

  • ADBC
  • ODBC
  • Go

SQL language

Lakehouse//RT runs SQL read queries in ANSI mode only.

Lakehouse//RT does not support the following:

  • Data types: The GEOGRAPHY and GEOMETRY data types.
  • Functions: AI functions, Python UDFs, spatial SQL functions, and XPath and XML functions.
  • Governance: Attribute-based access control (ABAC), including row-level security and column masking.

Lakehouse//RT is for read (SELECT) queries only. Write and ETL commands are not supported, including:

  • Write operations: INSERT, UPDATE, DELETE, MERGE, and CREATE TABLE AS SELECT (CTAS).
  • DDL: CREATE, ALTER, DROP, and other statements that create or modify objects.
  • Security statements: GRANT and REVOKE.
  • Scripting, stored procedures, temporary tables, and multi-statement transactions.
  • Delta Lake maintenance: OPTIMIZE, ANALYZE, VACUUM, and REFRESH.

Network security

Lakehouse//RT is not supported with the following network configurations:

Compliance

Compliance security profiles are not supported.

Known issues

  • Lakehouse//RT enforces stricter SQL type checking than other SQL warehouse types. Queries that mix types, such as comparing strings and numbers or using COALESCE with incompatible types, may behave differently or fail. To avoid unexpected results, use explicit CAST expressions.
  • Last updated on