What is data warehousing on Azure Databricks?

The Azure Databricks Lakehouse Platform provides a complete end-to-end data warehousing solution. The Databricks Lakehouse Platform is built on open standards and APIs. The Databricks Lakehouse combines the ACID transactions and data governance of enterprise data warehouses with the flexibility and cost-efficiency of data lakes. Databricks SQL describes the enterprise data warehouse built into the Azure Databricks Lakehouse Platform that provides general compute resources for business analytics. The core offering of Databricks SQL is a SQL warehouse.

What is data modeling on Azure Databricks?

The Azure Databricks organizes data stored with Delta Lake in cloud object storage with familiar relations like database schemas, tables, and views. Azure Databricks recommends a multi-layer approach to validating, cleansing, and transforming data for analytics. For more information, see the medallion architecture.

What is Databricks SQL?

Note

Databricks SQL is not available in Azure China regions.

Databricks SQL provides general compute resources for SQL queries, visualizations, and dashboards that are executed against the tables in the lakehouse. Within Databricks SQL, these queries, visualizations, and dashboards are developed and executed using SQL editor.

What is the SQL editor?

Use the built-in SQL editor to expore schemas and to write, share, and reuse queries using familiar SQL syntax. Regularly used SQL code can be saved as snippets for quick reuse, and query results can be cached to keep run times short. Additionally, query updates can be scheduled to automatically refresh, as well as to issue alerts when meaningful changes occur in the data. Databricks SQL also allows analysts to make sense of data through visualizations and drag-and-drop dashboards for quick ad-hoc exploratory analysis.

What are the available warehouse types?

Databricks SQL supports three warehouse types, each with different levels of performance and feature support.

Note

For pricing for each warehouse type and a detailed feature comparison, see Databricks SQL. To learn about the latest Databricks SQL features, see Databricks SQL release notes.

  • Serverless: Supports all features in the pro SQL warehouse type, as well as advanced Databricks SQL performance features. SQL warehouses run in the customer’s Azure Databricks account using serverless compute. See Serverless compute.
  • Pro: Supports additional Databricks SQL performance features (compared to classic) and supports all Databricks SQL functionality. The pro SQL warehouse type is enabled by default.
  • Classic: Supports entry level performance features and a limited set of Databricks SQL functionality. The classic SQL warehouse type is not enabled by default.

What are the warehouse type defaults?

  • Serverless SQL warehouses are enabled for the workspace: For both the UI or the API, the default SQL warehouse type is serverless.
  • Serverless SQL warehouses are enabled for the workspace: Using the UI, the default SQL warehouse type is pro. When creating a SQL warehouse in the workspace using the API with default parameters, the default SQL warehouse type is classic.

Databricks SQL and third party BI tools

Databricks SQL also supports many third party BI and visualization tools for your Azure Databricks data warehouse solution. See also Databricks and PowerBI and Databricks and Tableau.

Databricks SQL APIs

Databricks SQL also provides a robust API for your programming needs.

Databricks SQL administrative tasks

For information on enabling Databricks SQL, creating and managing SQL warehouses, managing users and data access, and other administrative tasks, see Databricks SQL administration.

Developer tools

You can use a variety of developer tools to run SQL commands and scripts and to browse database objects in Azure Databricks. See Use a SQL database tool.