Query LLMs and agents on Azure Databricks

Azure Databricks provides multiple ways to query large language models (LLMs), foundation models, and deployed agents. Choose an interactive UI, SQL, REST APIs, or client libraries depending on your workflow.

AI Playground

The AI Playground is a no-code chat environment in your Azure Databricks workspace for prompting and comparing LLMs. Use it to experiment with prompts, tune parameters like temperature and max tokens, and prototype tool-calling agents and question-answering bots side by side before moving to code.

Foundation models

Azure Databricks serves foundation models natively as model APIs (model services) through Unity AI Gateway. Every account user can call the pay-per-token models in the system.ai schema with no setup. Choose the model APIs that fit your needs:

• System-provided model APIs in system.ai: Models served natively by Azure Databricks, ready to query with the OpenAI-compatible SDK, native provider APIs, or ai_query. See Query model APIs (model services).
• Custom model APIs: Create your own when you need models beyond the defaults, or to distribute load and set up fallbacks.

Query agents

After you build and deploy an agent, query the agent from your application. Agents can be hosted on Databricks Apps or Model Serving endpoints. Azure Databricks supports three query methods:

• Databricks OpenAI client: Recommended for new applications, with native streaming and full feature support.
• OpenAI-compatible REST API: Language-agnostic, works with any platform that already speaks the OpenAI API.
• ai_query: Query legacy agents hosted on Model Serving endpoints from SQL.

Enrich data using AI Functions

AI Functions are built-in SQL functions that apply LLMs and other models to data stored in Azure Databricks. Run them from Databricks SQL, notebooks, Lakeflow pipelines, or Workflows to classify support tickets, extract entities from documents, summarize content, or translate text at batch scale.

Choose between two styles of functions:

• Task-specific functions like ai_classify, ai_extract, and ai_parse_document are optimized for a single task and use Azure Databricks-managed, research-backed systems.
• ai_query is the general-purpose function. Create your own prompt and choose any supported foundation model. See Use ai_query.

For an end-to-end example, see Analyze customer reviews using AI Functions.