Cribl connector reference

Important

This feature is in Beta. To use it, a workspace admin must turn on Lakeflow Connect for Cribl from the Previews page. See Manage Azure Databricks previews.

This page has reference information for the managed Cribl connector, including supported source tables and destination table schemas.

Supported source tables

The Cribl connector supports the following source tables, all under the default source schema. Worker Group-scoped tables ingest from every Stream, Edge, and Search Worker Group in the Workspace, unless you set group_allowlist.

Source table	Primary key	Description	Sync mode	Cursor field
worker_groups	product, id	Worker Groups and Fleets for the Stream, Edge, Search, and Outpost products.	Full refresh only
workers	product, id	Workers and Edge Nodes for the Stream and Edge products.	Full refresh only
health	host	Health status of the Workspace Leader.	Full refresh only
pipelines	group, id	Pipelines in each Worker Group.	Full refresh only
routes	group, id	Routing table in each Worker Group.	Full refresh only
sources	group, id	Source (input) configurations in each Worker Group.	Full refresh only
destinations	group, id	Destination (output) configurations in each Worker Group.	Full refresh only
parsers	group, id	Parser library entries in each Worker Group.	Full refresh only
breakers	group, id	Event Breaker rulesets in each Worker Group.	Full refresh only
grok_patterns	group, id	Grok pattern files in each Worker Group.	Full refresh only
schemas	group, id	Schema library entries in each Worker Group.	Full refresh only
collectors	group, id	Collector jobs in each Worker Group.	Full refresh only
database_connections	group, id	Database connections in each Worker Group.	Full refresh only
packs	group, id	Packs installed in each Worker Group.	Full refresh only
inputs_status	group, id	Health status and metrics for each Source in each Worker Group.	Full refresh only
outputs_status	group, id	Health status and metrics for each Destination in each Worker Group.	Full refresh only
functions	worker_group, id	Functions available in each Worker Group.	Full refresh only
workspaces	workspaceId	Workspaces in the Organization.	Full refresh only
api_credentials	clientId	API Credentials in the Organization. Client secrets are not ingested.	Full refresh only
audit_log	time, event_id	Leader audit log events.	Incremental	time
access_log	time, event_id	Leader API access log events.	Incremental	time

Connector options

The Cribl connector supports the following pipeline-level options. For usage, see Examples.

Option	Type	Scope	Required	Applies to	Description
control_plane_host	String	Pipeline	No	All tables except workspaces and api_credentials	Hostname of the Cribl.Cloud Workspace API, without the https:// scheme. Defaults to <workspace-id>-<organization-id>.cribl.cloud.
management_plane_host	String	Pipeline	No	workspaces, api_credentials	Hostname of the Cribl.Cloud management API, without the https:// scheme. Defaults to gateway.cribl.cloud.
group_allowlist	String	Pipeline	No	All Worker Group-scoped tables	Comma-separated list of Worker Group IDs to ingest. Defaults to all Worker Groups.
start_datetime	String	Pipeline	No	audit_log, access_log	Earliest log time to ingest on the first sync, in yyyy-MM-dd'T'HH:mm:ss.SSS'Z' format. Defaults to 365 days before the sync.

Destination table schemas

worker_groups

Primary key: product, id

Sync mode: Full refresh only

Field	Data type
id	STRING
name	STRING
description	STRING
type	STRING
tags	STRING
streamtags	ARRAY<STRING>
onPrem	BOOLEAN
provisioned	BOOLEAN
estimatedIngestRate	BIGINT
cloud	STRUCT<provider: STRING, region: STRING>
workerRemoteAccess	BOOLEAN
product	STRING
isFleet	BOOLEAN
isSearch	BOOLEAN
collectorsHaEnabled	BOOLEAN
configVersion	STRING
workerCount	BIGINT
deployingWorkerCount	BIGINT
incompatibleWorkerCount	BIGINT
maxWorkerAge	STRING
upgradeVersion	STRING
inherits	STRING
git	STRUCT<commit: STRING, localChanges: BIGINT, log: ARRAY<STRUCT<author_email: STRING, author_name: STRING, date: STRING, hash: STRING, message: STRING, short: STRING>>>
lookupDeployments	ARRAY<STRUCT<context: STRING, lookups: ARRAY<STRUCT<deployedVersion: STRING, file: STRING, version: STRING>>>>

workers

Primary key: product, id

Sync mode: Full refresh only

Field	Data type
id	STRING
status	STRING
group	STRING
info	STRUCT<apiPort: BIGINT, apiScheme: STRING, architecture: STRING, aws: STRUCT<enabled: BOOLEAN, instanceId: STRING, region: STRING, tags: MAP<STRING, STRING>, type: STRING, zone: STRING>, azure: STRUCT<enabled: BOOLEAN, hostname: STRING, instanceId: STRING, name: STRING, region: STRING, resourceGroup: STRING, subscriptionId: STRING, tags: MAP<STRING, STRING>, type: STRING, vmId: STRING, zone: STRING>, conn_ip: STRING, cpus: BIGINT, cribl: STRUCT<config: STRUCT<apiCredentialsRev: STRING, featuresRev: STRING, hbPeriodSeconds: BIGINT, logStreamEnv: STRING, policyRev: STRING, teamsRev: STRING, usersRev: STRING, version: STRING>, deploymentId: STRING, disableSNIRouting: BOOLEAN, distMode: STRING, edgeNodes: BIGINT, group: STRING, guid: STRING, installType: STRING, lookupVersions: MAP<STRING, MAP<STRING, STRING>>, master: STRUCT<host: STRING, port: BIGINT, servername: STRING, tls: BOOLEAN>, overlayId: STRING, pid: BIGINT, socksEnabled: BOOLEAN, startTime: BIGINT, tags: ARRAY<STRING>, version: STRING>, env: MAP<STRING, STRING>, freeDiskSpace: BIGINT, hostOs: STRUCT<addresses: ARRAY<STRING>, enabled: BOOLEAN, id: STRING, version: STRING>, hostname: STRING, isCaptain: BOOLEAN, isSaasWorker: BOOLEAN, kube: STRUCT<enabled: BOOLEAN, namespace: STRING, node: STRING, owner: STRUCT<kind: STRING, name: STRING>, pod: STRING, source: STRING>, localTime: BIGINT, metadata: STRUCT<aws: STRUCT<enabled: BOOLEAN, instanceId: STRING, region: STRING, tags: MAP<STRING, STRING>, type: STRING, zone: STRING>, azure: STRUCT<enabled: BOOLEAN, hostname: STRING, instanceId: STRING, name: STRING, region: STRING, resourceGroup: STRING, subscriptionId: STRING, tags: MAP<STRING, STRING>, type: STRING, vmId: STRING, zone: STRING>, hostOs: STRUCT<addresses: ARRAY<STRING>, enabled: BOOLEAN, id: STRING, version: STRING>, kube: STRUCT<enabled: BOOLEAN, namespace: STRING, node: STRING, owner: STRUCT<kind: STRING, name: STRING>, pod: STRING, source: STRING>, os: STRUCT<addresses: ARRAY<STRING>, enabled: BOOLEAN, id: STRING, version: STRING>>, node: STRING, os: STRUCT<addresses: ARRAY<STRING>, enabled: BOOLEAN, id: STRING, version: STRING>, outpost: STRUCT<groupname: STRING, guid: STRING, host: STRING>, platform: STRING, release: STRING, totalDiskSpace: BIGINT, totalmem: BIGINT>
workerProcesses	BIGINT
lastMsgTime	BIGINT
firstMsgTime	BIGINT
deployable	BOOLEAN
disconnected	BOOLEAN
connectionProtocol	STRING
product	STRING
type	STRING
offlineDurationMs	BIGINT
provisioningTokenId	STRING
metadata	STRUCT<aws: STRUCT<enabled: BOOLEAN, instanceId: STRING, region: STRING, tags: MAP<STRING, STRING>, type: STRING, zone: STRING>, azure: STRUCT<enabled: BOOLEAN, hostname: STRING, instanceId: STRING, name: STRING, region: STRING, resourceGroup: STRING, subscriptionId: STRING, tags: MAP<STRING, STRING>, type: STRING, vmId: STRING, zone: STRING>, hostOs: STRUCT<addresses: ARRAY<STRING>, enabled: BOOLEAN, id: STRING, version: STRING>, kube: STRUCT<enabled: BOOLEAN, namespace: STRING, node: STRING, owner: STRUCT<kind: STRING, name: STRING>, pod: STRING, source: STRING>, os: STRUCT<addresses: ARRAY<STRING>, enabled: BOOLEAN, id: STRING, version: STRING>>
lastMetrics	MAP<STRING, VARIANT>
nodeUpgradeStatus	STRUCT<active: BIGINT, failed: BIGINT, skipped: BIGINT, state: BIGINT, timestamp: BIGINT>
workers	STRUCT<count: BIGINT>

Note

This table does not support SCD type 2 because it has VARIANT columns.

health

Primary key: host

Sync mode: Full refresh only

Field	Data type
status	STRING
startTime	BIGINT
overlay	STRUCT<state: STRING>
role	STRING
isCaptain	BOOLEAN
host	STRING

pipelines

Primary key: group, id

Sync mode: Full refresh only

Field	Data type
group	STRING
id	STRING
conf	STRUCT<__template_streamtags: STRING, asyncFuncTimeout: BIGINT, description: STRING, functions: ARRAY<STRUCT<conf: MAP<STRING, VARIANT>, description: STRING, disabled: BOOLEAN, filter: STRING, final: BOOLEAN, groupId: STRING, id: STRING>>, groups: MAP<STRING, STRUCT<description: STRING, disabled: BOOLEAN, name: STRING>>, output: STRING, pack: BOOLEAN, streamtags: ARRAY<STRING>>
conf_description	STRING
conf_output	STRING

Note

This table does not support SCD type 2 because it has VARIANT columns.

routes

Primary key: group, id

Sync mode: Full refresh only

Field	Data type
group	STRING
id	STRING
routes	ARRAY<STRUCT<autoParse: BOOLEAN, clones: ARRAY<MAP<STRING, STRING>>, context: STRING, description: STRING, disabled: BOOLEAN, enableOutputExpression: BOOLEAN, filter: STRING, final: BOOLEAN, groupId: STRING, id: STRING, name: STRING, output: STRING, outputExpression: STRING, pipeline: STRING, targetContext: STRING>>
comments	ARRAY<STRUCT<comment: STRING, groupId: STRING, id: STRING, index: BIGINT>>
groups	MAP<STRING, STRUCT<description: STRING, index: BIGINT, name: STRING>>

sources

Primary key: group, id

Sync mode: Full refresh only

Field	Data type
group	STRING
id	STRING
type	STRING
pipeline	STRING
disabled	BOOLEAN
fullFidelity	BOOLEAN
sendToRoutes	BOOLEAN
pqEnabled	BOOLEAN
streamtags	ARRAY<STRING>
status	STRUCT<error: STRUCT<details: MAP<STRING, VARIANT>, message: STRING>, health: STRING, metrics: MAP<STRING, VARIANT>, pq: STRUCT<error: STRUCT<details: MAP<STRING, VARIANT>, message: STRING>, health: BIGINT, metrics: MAP<STRING, VARIANT>, timestamp: BIGINT>, timestamp: BIGINT, useStatusFromLB: BOOLEAN>
notifications	ARRAY<STRUCT<mode: STRING, templateTargetPairs: ARRAY<STRUCT<targetId: STRING, templateId: STRING>>>>
raw	VARIANT

Note

This table does not support SCD type 2 because it has VARIANT columns.

destinations

Primary key: group, id

Sync mode: Full refresh only

Field	Data type
group	STRING
id	STRING
type	STRING
pipeline	STRING
environment	STRING
description	STRING
defaultId	STRING
systemFields	ARRAY<STRING>
streamtags	ARRAY<STRING>
status	STRUCT<error: STRUCT<details: MAP<STRING, VARIANT>, message: STRING>, health: STRING, metrics: MAP<STRING, VARIANT>, pq: STRUCT<error: STRUCT<details: MAP<STRING, VARIANT>, message: STRING>, health: BIGINT, metrics: MAP<STRING, VARIANT>, timestamp: BIGINT>, timestamp: BIGINT, useStatusFromLB: BOOLEAN>
notifications	ARRAY<STRUCT<mode: STRING, templateTargetPairs: ARRAY<STRUCT<targetId: STRING, templateId: STRING>>>>
raw	VARIANT

Note

This table does not support SCD type 2 because it has VARIANT columns.

parsers

Primary key: group, id

Sync mode: Full refresh only

Field	Data type
group	STRING
id	STRING
type	STRING
lib	STRING
description	STRING
tags	STRING
fields	ARRAY<STRING>
delimChar	STRING
escapeChar	STRING
nullValue	STRING
quoteChar	STRING
regex	STRING
raw	VARIANT

Note

This table does not support SCD type 2 because it has VARIANT columns.

breakers

Primary key: group, id

Sync mode: Full refresh only

Field	Data type
group	STRING
id	STRING
lib	STRING
description	STRING
tags	STRING
rules	ARRAY<STRUCT<__template_timestampTimezone: STRING, cleanFields: BOOLEAN, condition: STRING, delimiter: STRING, delimiterRegex: STRING, disabled: BOOLEAN, escapeChar: STRING, eventBreakerRegex: STRING, fields: ARRAY<STRUCT<name: STRING, value: STRING>>, fieldsLineRegex: STRING, fieldsToRemove: ARRAY<STRING>, hasHeaderRow: BOOLEAN, headerLineRegex: STRING, index: BIGINT, jsonArrayField: STRING, jsonExtractAll: BOOLEAN, jsonTimeField: STRING, maxEventBytes: BIGINT, name: STRING, nullFieldVal: STRING, parentFieldsToCopy: ARRAY<STRING>, parser: VARIANT, parserEnabled: BOOLEAN, quoteChar: STRING, shouldUseDataRaw: BOOLEAN, timeField: STRING, timestamp: STRUCT<format: STRING, length: BIGINT, type: STRING>, timestampAnchorRegex: STRING, timestampEarliest: STRING, timestampLatest: STRING, timestampSourceField: STRING, timestampTimezone: STRING, type: STRING>>
minRawLength	BIGINT
raw	VARIANT

Note

This table does not support SCD type 2 because it has VARIANT columns.

grok_patterns

Primary key: group, id

Sync mode: Full refresh only

Field	Data type
group	STRING
id	STRING
content	STRING
size	BIGINT
tags	STRING

schemas

Primary key: group, id

Sync mode: Full refresh only

Field	Data type
group	STRING
id	STRING
description	STRING
schema	STRING
tags	STRING

collectors

Primary key: group, id

Sync mode: Full refresh only

Field	Data type
group	STRING
id	STRING
type	STRING
description	STRING
environment	STRING
ttl	STRING
savedQueryId	STRING
__template_streamtags	STRING
resumeOnBoot	BOOLEAN
workerAffinity	BOOLEAN
ignoreGroupJobsLimit	BOOLEAN
removeFields	ARRAY<STRING>
streamtags	ARRAY<STRING>
collector	STRUCT<conf: VARIANT, destructive: BOOLEAN, encoding: STRING, type: STRING>
executor	STRUCT<conf: VARIANT, storeTaskResults: BOOLEAN, type: STRING>
input	STRUCT<breakerRulesets: ARRAY<STRING>, metadata: ARRAY<STRUCT<name: STRING, value: STRING>>, output: STRING, pipeline: STRING, preprocess: STRUCT<args: ARRAY<STRING>, command: STRING, disabled: BOOLEAN>, sendToRoutes: BOOLEAN, staleChannelFlushMs: DOUBLE, throttleRatePerSec: STRING, type: STRING>
schedule	STRUCT<cronSchedule: STRING, enabled: BOOLEAN, maxConcurrentRuns: BIGINT, resumeMissed: BOOLEAN, run: MAP<STRING, VARIANT>, skippable: BOOLEAN>
history	VARIANT
savedState	MAP<STRING, STRUCT<data: MAP<STRING, VARIANT>>>
notifications	ARRAY<STRUCT<condition: STRING, conf: VARIANT, disabled: BOOLEAN, group: STRING, id: STRING, metadata: ARRAY<STRUCT<name: STRING, value: STRING>>, mode: STRING, pack: STRING, targetConfigs: ARRAY<STRUCT<conf: VARIANT, id: STRING>>, targets: ARRAY<STRING>, templateTargetPairs: ARRAY<STRUCT<targetId: STRING, templateId: STRING>>>>

Note

This table does not support SCD type 2 because it has VARIANT columns.

database_connections

Primary key: group, id

Sync mode: Full refresh only

Field	Data type
group	STRING
id	STRING
databaseType	STRING
authType	STRING
description	STRING
tags	STRING
connectionTimeout	BIGINT
requestTimeout	BIGINT
tls	STRUCT<caPath: STRING, certPath: STRING, certificateName: STRING, disabled: BOOLEAN, maxVersion: STRING, minVersion: STRING, privKeyPath: STRING, rejectUnauthorized: BOOLEAN, servername: STRING>

packs

Primary key: group, id

Sync mode: Full refresh only

Field	Data type
group	STRING
id	STRING
source	STRING
version	STRING
description	STRING
displayName	STRING
author	STRING
tags	VARIANT
raw	VARIANT

Note

This table does not support SCD type 2 because it has VARIANT columns.

inputs_status

Primary key: group, id

Sync mode: Full refresh only

Field	Data type
group	STRING
id	STRING
type	STRING
status	STRUCT<error: STRUCT<details: MAP<STRING, VARIANT>, message: STRING>, health: STRING, healthCounts: STRUCT<Green: BIGINT, Red: BIGINT, Unknown: BIGINT, Yellow: BIGINT>, metrics: MAP<STRING, VARIANT>, pq: STRUCT<error: STRUCT<details: MAP<STRING, VARIANT>, message: STRING>, health: STRING, healthCounts: STRUCT<Green: BIGINT, Red: BIGINT, Unknown: BIGINT, Yellow: BIGINT>, metrics: MAP<STRING, VARIANT>, timestamp: BIGINT>, timestamp: BIGINT, useStatusFromLB: BOOLEAN>
health	STRING
status_timestamp	BIGINT

Note

This table does not support SCD type 2 because it has VARIANT columns.

outputs_status

Primary key: group, id

Sync mode: Full refresh only

Field	Data type
group	STRING
id	STRING
type	STRING
status	STRUCT<error: STRUCT<details: MAP<STRING, VARIANT>, message: STRING>, health: STRING, healthCounts: STRUCT<Green: BIGINT, Red: BIGINT, Unknown: BIGINT, Yellow: BIGINT>, metrics: MAP<STRING, VARIANT>, pq: STRUCT<error: STRUCT<details: MAP<STRING, VARIANT>, message: STRING>, health: STRING, healthCounts: STRUCT<Green: BIGINT, Red: BIGINT, Unknown: BIGINT, Yellow: BIGINT>, metrics: MAP<STRING, VARIANT>, timestamp: BIGINT>, timestamp: BIGINT, useStatusFromLB: BOOLEAN>
health	STRING
status_timestamp	BIGINT

Note

This table does not support SCD type 2 because it has VARIANT columns.

functions

Primary key: worker_group, id

Sync mode: Full refresh only

Field	Data type
worker_group	STRING
id	STRING
name	STRING
version	STRING
group	STRING
disabled	BOOLEAN
handleSignals	BOOLEAN
sync	BOOLEAN
__filename	STRING
loadTime	DOUBLE
modTime	DOUBLE
asyncTimeout	DOUBLE
cribl_version	STRING
schema	MAP<STRING, VARIANT>
uischema	MAP<STRING, VARIANT>

Note

This table does not support SCD type 2 because it has VARIANT columns.

workspaces

Primary key: workspaceId

Sync mode: Full refresh only

Field	Data type
workspaceId	STRING
region	STRING
leaderFQDN	STRING
state	STRING
alias	STRING
description	STRING
tags	ARRAY<STRING>

api_credentials

Primary key: clientId

Sync mode: Full refresh only

Field	Data type
name	STRING
description	STRING
enabled	BOOLEAN
organizationId	STRING
clientId	STRING
roles	VARIANT
organizationRole	STRING
createdBy	STRING
createdDate	TIMESTAMP
lastUpdatedBy	STRING
lastUpdatedDate	TIMESTAMP
ipAllowlist	ARRAY<STRING>

Note

This table does not support SCD type 2 because it has VARIANT columns.

audit_log

Primary key: time, event_id

Cursor field: time

Field	Data type
time	TIMESTAMP
action	STRING
type	STRING
user	STRING
principal	STRING
id	STRING
requestId	STRING
group	STRING
event_id	BIGINT
raw	VARIANT

Note

This table does not support SCD type 2 because it has VARIANT columns.

access_log

Primary key: time, event_id

Cursor field: time

Field	Data type
time	TIMESTAMP
src	STRING
user	STRING
method	STRING
url	STRING
status	BIGINT
message	STRING
response_time	BIGINT
requestId	STRING
cid	STRING
stats	VARIANT
event_id	BIGINT
raw	VARIANT

Note

This table does not support SCD type 2 because it has VARIANT columns.