本页介绍如何向 Azure Databricks 上 Unity Gateway 提供的 Gemini 基础模型发送音频和视频输入。 你可以使用聊天补全 API 或 Google Gemini API,通过 URL 或 base64 编码的内联数据提供媒体。
要求
输入方法
可以使用两种方法提供音频和视频输入:
- URL:将可公开访问的 URL 传递给媒体文件。 对于视频,还支持 YouTube URL。
-
Base64 内联数据:将文件编码为 base64 字符串,并将其作为数据 URI 传递(例如)。
data:video/mp4;base64,<encoded_data>
注释
以下示例基于 Unity Gateway 和模型服务。 如果使用模型服务终结点而不是模型服务,请将模型服务名称替换为终结点名称。 请参阅 Databricks 基础模型 API 支持的模型详细列表 ,了解可用的基础模型及其模型服务和端点名称。
聊天补全 API
聊天完成 API 允许传递视频和音频输入。 在video_url数组中使用audio_url和messages内容类型来传递媒体输入。 每个内容项都包含一个 url 接受 Web URL 或 base64 数据 URI 的字段。
视频输入
Python
import os
import base64
from openai import OpenAI
DATABRICKS_TOKEN = os.environ.get('DATABRICKS_TOKEN')
DATABRICKS_BASE_URL = os.environ.get('DATABRICKS_BASE_URL')
client = OpenAI(
api_key=DATABRICKS_TOKEN,
base_url=DATABRICKS_BASE_URL
)
# Encode a local video file as base64
with open("video.mp4", "rb") as f:
video_b64 = base64.standard_b64encode(f.read()).decode("utf-8")
response = client.chat.completions.create(
model="system.ai.gemini-3-1-pro",
messages=[{
"role": "user",
"content": [
{"type": "text", "text": "Summarize what happens in these videos."},
{
"type": "video_url",
"video_url": {"url": "https://example.com/sample-video.mp4"}
},
{
"type": "video_url",
"video_url": {"url": f"data:video/mp4;base64,{video_b64}"}
},
]
}],
max_tokens=1024
)
print(response.choices[0].message.content)
REST API
curl \
-u token:$DATABRICKS_TOKEN \
-X POST \
-H "Content-Type: application/json" \
-d '{
"model": "system.ai.gemini-3-1-pro",
"messages": [{
"role": "user",
"content": [
{"type": "text", "text": "Summarize what happens in these videos."},
{
"type": "video_url",
"video_url": {"url": "https://example.com/sample-video.mp4"}
},
{
"type": "video_url",
"video_url": {"url": "data:video/mp4;base64,<base64_encoded_data>"}
}
]
}],
"max_tokens": 1024
}' \
https://<workspace_host>.databricks.com/ai-gateway/mlflow/v1/chat/completions
音频输入
Python
import os
import base64
from openai import OpenAI
DATABRICKS_TOKEN = os.environ.get('DATABRICKS_TOKEN')
DATABRICKS_BASE_URL = os.environ.get('DATABRICKS_BASE_URL')
client = OpenAI(
api_key=DATABRICKS_TOKEN,
base_url=DATABRICKS_BASE_URL
)
# Encode a local audio file as base64
with open("audio.mp3", "rb") as f:
audio_b64 = base64.standard_b64encode(f.read()).decode("utf-8")
response = client.chat.completions.create(
model="system.ai.gemini-3-1-pro",
messages=[{
"role": "user",
"content": [
{"type": "text", "text": "Transcribe this audio and summarize the key points."},
{
"type": "audio_url",
"audio_url": {"url": "https://example.com/sample-audio.mp3"}
},
{
"type": "audio_url",
"audio_url": {"url": f"data:audio/mp3;base64,{audio_b64}"}
},
]
}],
max_tokens=1024
)
print(response.choices[0].message.content)
REST API
curl \
-u token:$DATABRICKS_TOKEN \
-X POST \
-H "Content-Type: application/json" \
-d '{
"model": "system.ai.gemini-3-1-pro",
"messages": [{
"role": "user",
"content": [
{"type": "text", "text": "Transcribe this audio and summarize the key points."},
{
"type": "audio_url",
"audio_url": {"url": "https://example.com/sample-audio.mp3"}
},
{
"type": "audio_url",
"audio_url": {"url": "data:audio/mp3;base64,<base64_encoded_data>"}
}
]
}],
"max_tokens": 1024
}' \
https://<workspace_host>.databricks.com/ai-gateway/mlflow/v1/chat/completions
谷歌双子座API
使用 Google Gemini API 将媒体作为inlineData数组中的 fileData (base64 编码)或 parts (URL 引用)传递。
视频输入
Python
from google import genai
from google.genai import types
import base64
import os
DATABRICKS_TOKEN = os.environ.get('DATABRICKS_TOKEN')
client = genai.Client(
api_key="databricks",
http_options=types.HttpOptions(
base_url="https://example.staging.cloud.databricks.com/ai-gateway/gemini",
headers={
"Authorization": f"Bearer {DATABRICKS_TOKEN}",
},
),
)
# Encode a local video file as base64
with open("video.mp4", "rb") as f:
video_b64 = base64.standard_b64encode(f.read()).decode("utf-8")
response = client.models.generate_content(
model="databricks-gemini-3-1-pro",
contents=[
types.Content(
role="user",
parts=[
types.Part(text="Summarize what happens in these videos."),
types.Part(
file_data=types.FileData(
mime_type="video/mp4",
file_uri="https://example.com/sample-video.mp4",
)
),
types.Part(
inline_data=types.Blob(
mime_type="video/mp4",
data=video_b64,
)
),
],
),
],
config=types.GenerateContentConfig(
max_output_tokens=1024,
),
)
print(response.text)
REST API
curl \
-u token:$DATABRICKS_TOKEN \
-X POST \
-H "Content-Type: application/json" \
-d '{
"contents": [{
"role": "user",
"parts": [
{"text": "Summarize what happens in these videos."},
{
"fileData": {
"mimeType": "video/mp4",
"fileUri": "https://example.com/sample-video.mp4"
}
},
{
"inlineData": {
"mimeType": "video/mp4",
"data": "<base64_encoded_data>"
}
}
]
}]
}' \
https://<workspace_host>.databricks.com/ai-gateway/gemini/v1beta/models/databricks-gemini-3-1-pro:generateContent
音频输入
Python
from google import genai
from google.genai import types
import base64
import os
DATABRICKS_TOKEN = os.environ.get('DATABRICKS_TOKEN')
client = genai.Client(
api_key="databricks",
http_options=types.HttpOptions(
base_url="https://example.staging.cloud.databricks.com/ai-gateway/gemini",
headers={
"Authorization": f"Bearer {DATABRICKS_TOKEN}",
},
),
)
# Encode a local audio file as base64
with open("audio.mp3", "rb") as f:
audio_b64 = base64.standard_b64encode(f.read()).decode("utf-8")
response = client.models.generate_content(
model="databricks-gemini-3-1-pro",
contents=[
types.Content(
role="user",
parts=[
types.Part(text="Transcribe this audio and summarize the key points."),
types.Part(
file_data=types.FileData(
mime_type="audio/mp3",
file_uri="https://example.com/sample-audio.mp3",
)
),
types.Part(
inline_data=types.Blob(
mime_type="audio/mp3",
data=audio_b64,
)
),
],
),
],
config=types.GenerateContentConfig(
max_output_tokens=1024,
),
)
print(response.text)
REST API
curl \
-u token:$DATABRICKS_TOKEN \
-X POST \
-H "Content-Type: application/json" \
-d '{
"contents": [{
"role": "user",
"parts": [
{"text": "Transcribe this audio and summarize the key points."},
{
"fileData": {
"mimeType": "audio/mp3",
"fileUri": "https://example.com/sample-audio.mp3"
}
},
{
"inlineData": {
"mimeType": "audio/mp3",
"data": "<base64_encoded_data>"
}
}
]
}]
}' \
https://<workspace_host>.databricks.com/ai-gateway/gemini/v1beta/models/databricks-gemini-3-1-pro:generateContent
支持的模型
以下 Gemini 按令牌付费的基础模型支持音频和视频输入。 有关地区可用性,请参阅 Databricks Foundation 模型 API 支持的详细模型列表 。
databricks-gemini-3-1-prodatabricks-gemini-3-1-flash-litedatabricks-gemini-3-flash
局限性
- 音频和视频输入仅适用于 Gemini 按令牌付费的基础模型。 不支持预配置的吞吐量端点。
- 单个请求中可以包含多个音频或视频输入,但大型文件会增加延迟和令牌使用量。