Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
Important
This feature is in Beta. Workspace admins can control access to this feature from the Previews page. See Manage Azure Databricks previews.
The managed OpenAI connector in Lakeflow Connect ingests organization administration data, such as users, projects, API keys, usage, costs, and audit logs, from OpenAI into Azure Databricks.
Feature availability
| Feature | Availability |
|---|---|
| UI-based pipeline authoring |  Supported |
| API-based pipeline authoring | Supported |
| Declarative Automation Bundles | Supported |
| Incremental ingestion | Supported |
| Unity Catalog governance | Supported |
| Orchestration using Databricks Workflows | Supported |
| API-based column selection and deselection | Supported |
| API-based row filtering |  Not supported |
| SCD type 2 | Supported Applies to the mutable entity tables. The append-only audit_logs, usage_*, and costs tables record immutable events and daily buckets. |
| Automated schema evolution: New and deleted columns | Supported |
| Automated schema evolution: Data type changes | Not supported |
| Automated schema evolution: Column renames | Not supported Requires a full refresh. |
Authentication methods
| Authentication method | Availability |
|---|---|
| OAuth U2M | Not supported |
| OAuth M2M | Not supported |
| Basic authentication (username/password) | Not supported |
| Basic authentication (API key) | Supported OpenAI Admin API key. |
What to know before you start
| Topic | Why it matters |
|---|---|
| Azure Databricks user persona | The workflow depends on your Azure Databricks user persona:
|
| Authentication method | The steps to create a connection depend on the authentication method you choose. |
| Interface | The steps to create a pipeline depend on the interface. |
| Ingestion frequency | The pipeline schedule depends on your latency and cost requirements. |
| Common patterns | Depending on your ingestion needs, the pipeline might use configurations like history tracking, column selection, and row filtering. Supported configurations vary by connector. See Feature availability. |
Ingest from OpenAI in 3 steps
- Configure OpenAI for ingestion (Admins) — Set up OpenAI to authenticate with Azure Databricks.
- Create a Unity Catalog connection (Admins) — Create a connection in Catalog Explorer to store credentials for Databricks to authenticate with OpenAI.
- Create an ingestion pipeline (Admins or non-admins) — Select any supported interface and create a pipeline from an existing connection.