Extended network traffic logs

Important

This feature is in Private Preview. To try it, reach out to your Azure Databricks contact.

Extended network traffic logs record outbound network traffic from Azure Databricks serverless compute. By default, the outbound network access events system table records only the requests that serverless egress controls deny. With extended network traffic logs, it also records the requests that were allowed, so you can see where your serverless workloads connect and troubleshoot connectivity.

Extended network traffic logs also add a flow logs table that records how much data serverless compute sends to and receives from each destination IP address. The two capabilities use the following system tables:

Capability	System table	Description
Allowed outbound network access events	system.access.outbound_network	Adds allowed outbound requests to the existing table of denied requests.
Outbound traffic flow logs	system.access.outbound_traffic	Records bytes sent and received per destination IP address, aggregated into one-minute intervals.

Outbound traffic flow logs only record traffic routed through a private network gateway.

To query the logs, see Query extended network traffic logs.

Use cases

Use extended network traffic logs to:

• Investigate network activity: See which destinations your serverless workloads accessed, when, and from which workspace.
• Refine network policies: Find destinations that are rarely accessed and decide whether to remove them from your serverless egress allowlists.
• Troubleshoot connectivity: Determine whether serverless egress controls allowed or denied a request.
• Investigate networking costs: Identify cross-region, internet, and private destinations accessed by serverless workloads.
• Optimize storage connectivity: Find frequently accessed storage destinations that might benefit from private connectivity or from being in the same region as your workloads.

• Measure traffic volumes: See how much data serverless compute sends to and receives from each destination through a private network gateway.
• Detect unusual activity: Find destinations with unexpectedly high traffic volumes.

How it works

Allowed network access events

Extended network traffic logs add ALLOWED events to the existing system.access.outbound_network table. This table records outbound network access events from serverless workloads, including SQL warehouses, jobs, Lakeflow pipelines, Databricks Apps, and Model Serving endpoints. Allowed events cover requests to cloud storage, internet destinations, and resources reached through private connectivity. For the complete table schema, see Network access events system table reference.

The access_type column indicates the outcome of each event:

Value	Description
ALLOWED	Outbound network access was allowed. Available only with extended network traffic logs.
DROP	Outbound network access was denied by serverless egress controls in enforced mode.
DRY_RUN_DENIAL	Outbound network access would have been denied by serverless egress controls, but was allowed because the policy was in dry-run mode.

The access_type and destination_type values recorded for a request depend on the destination and your serverless egress policy mode:

Network request	Unrestricted egress	Dry-run mode	Enforced mode
Allowed storage access	ALLOWED / DNS	Not logged	ALLOWED / STORAGE
Storage access that violates the policy	Not applicable	DRY_RUN_DENIAL / DNS	DROP / STORAGE
Allowed access to VNet resources	ALLOWED / DNS	ALLOWED / DNS	ALLOWED / DNS
Access to VNet resources that violates the policy	Not applicable	DRY_RUN_DENIAL / DNS	DROP / DNS
Allowed internet access	ALLOWED / DNS	ALLOWED / DNS	ALLOWED / DNS
Internet access that violates the policy	Not applicable	DRY_RUN_DENIAL / DNS	DROP / DNS

For allowed DNS events, destination contains the domain name or storage hostname. For allowed STORAGE events, destination contains the storage hostname and path. The resolved destination IP addresses are in dns_event.rdata or storage_event.rdata. These fields are empty for denied events.

Outbound traffic flow logs

The system.access.outbound_traffic table records aggregated traffic volumes, not individual requests. Azure Databricks groups traffic into fixed one-minute intervals that start and end on a minute boundary. Within each interval, traffic with the same workspace, destination IP address, and connection method is combined into one record, with the bytes sent and received summed.

For example, traffic between 12:05:00 and 12:06:00 produces a record with period_start_time = 12:05:00 and period_end_time = 12:06:00. The start time is inclusive and the end time is exclusive. These timestamps mark the interval boundaries, not the times of the first and last packets. Intervals without traffic don't produce records.

The system.access.outbound_traffic table has the following schema:

Column	Data type	Description
event_id	string	Unique identifier for the aggregated traffic record.
account_id	string	ID of the Azure Databricks account where the traffic originated.
workspace_id	string	ID of the workspace where the traffic originated.
destination_ip	string	Destination IP address.
received_bytes	bigint	Total bytes received by serverless compute during the interval.
sent_bytes	bigint	Total bytes sent by serverless compute during the interval.
connection_type	string	Type of connectivity used. The value is PRIVATE.
connection_method	string	Connectivity method used. The value is PRIVATE_NETWORK_GATEWAY.
period_start_time	timestamp	Inclusive start of the one-minute interval.
period_end_time	timestamp	Exclusive end of the one-minute interval.

How the tables relate

system.access.outbound_network tells you the destination hostname or storage location of a request. system.access.outbound_traffic tells you the destination IP address, the connection method, and how many bytes were transferred.

To correlate them, match the account ID, workspace ID, and resolved destination IP address, and check that the access event's event_time falls within the flow log's interval. The two tables have independent event_id values, so don't join on event_id.

These matches are correlations based on IP address and time. They don't link an access event to a specific network connection. When several hostnames resolve to the same IP address, each hostname matches the same flow records.

Limitations

Extended network traffic logs have the following limitations:

• Allowed events aren't logged for external model access from Model Serving endpoints.
• Allowed events aren't logged for storage access from workspaces whose serverless egress controls are in dry-run mode.
• Allowed events aren't logged for access to workspace-internal storage, such as MLflow and dbt artifacts.
• Allowed events aren't logged for access to OpenSharing provider storage.
• Allowed traffic from workloads that run for less than about one minute is less likely to be logged, especially for scheduled, non-interactive workloads and for workspaces with unrestricted egress policies.

• Outbound traffic flow logs only record traffic routed through a private network gateway. Traffic that uses other private connectivity or public internet egress isn't logged.
• Outbound traffic flow logs don't record traffic from Agent Bricks, Unity Gateway, or Git folders.

These limitations don't affect how serverless egress controls are enforced or how denied requests are logged.

Frequently asked questions

The following questions cover common behavior you might see in extended network traffic logs.

Do I need serverless egress controls to use extended network traffic logs?

No. Allowed outbound network access events are recorded for workspaces with unrestricted serverless egress policies too.

Why do network access events appear after a workload finishes?

Azure Databricks might continue to access storage after a workload finishes, for example to maintain Delta logs or metadata. These requests can generate network access events when no user workloads are running. The events stop after the serverless compute instance terminates.

Why do I see unfamiliar hostnames in my network access logs?

Some logged destinations are Azure Databricks-managed resources used for platform operations:

Hostname pattern	Purpose
databricks-prewarm-datasets*	Improves serverless compute startup times.
dbstorage*	Azure Databricks default storage.
jaeger-collector-privileged-worker*	System health tracing.
system-tables*	Access to default sample datasets.

If you see other hostnames you don't recognize, contact your Azure Databricks account team.

How long does it take for network traffic logs to appear?

Most outbound network access events appear within 15 minutes of the request. Latency can vary, and Azure Databricks doesn't provide a latency service level agreement (SLA) for these logs.

Why can't I find a matching flow log for an allowed network access event?

Flow logs only record traffic routed through a private network gateway, so allowed requests that use other connectivity have no matching flow log. A missing flow log doesn't mean the request failed or that no data was transferred.