Cribl connector

Important

This feature is in Beta. To use it, a workspace admin must turn on Lakeflow Connect for Cribl from the Previews page. See Manage Azure Databricks previews.

The managed Cribl connector in Lakeflow Connect ingests configuration, worker status, organization metadata, and Leader audit and access logs from Cribl.Cloud into Azure Databricks.

Feature availability

Feature	Availability
UI-based pipeline authoring	Supported
API-based pipeline authoring	Supported
Declarative Automation Bundles	Supported
Incremental ingestion	Supported
Unity Catalog governance	Supported
Orchestration using Databricks Workflows	Supported
API-based column selection and deselection	Supported
API-based row filtering	Not supported
SCD Type 2	Supported Tables with VARIANT columns (workers, pipelines, sources, destinations, parsers, breakers, collectors, packs, inputs_status, outputs_status, functions, api_credentials, audit_log, access_log) do not support SCD type 2.
Automated schema evolution: New and deleted columns	Supported
Automated schema evolution: Data type changes	Not supported
Automated schema evolution: Column renames	Not supported Requires a full refresh.

Authentication methods

Authentication method	Availability
OAuth U2M	Not supported
OAuth M2M	Supported Client ID and Client Secret from a Cribl.Cloud API Credential.
Basic authentication (username/password)	Not supported

Ingest from Cribl.Cloud in 3 steps

Note

Before starting, review the Azure Databricks user persona, supported interfaces, ingestion frequency, and common patterns.

1. Configure Cribl.Cloud for ingestion (Admins): Set up Cribl.Cloud to authenticate with Azure Databricks.
2. Create a Unity Catalog connection (Admins): Create a connection in Catalog Explorer to store credentials for Databricks to authenticate with Cribl.Cloud.
3. Create an ingestion pipeline (Admins or non-admins): Select any supported interface and create a pipeline from an existing connection.