Export Langfuse traces to Azure Databricks

Configure Langfuse to send OTel spans to the Azure Databricks OTLP endpoint. Traces are stored in Unity Catalog tables alongside your other MLflow traces, where you can query and compare them with SQL or view them in the MLflow UI.

Consolidating traces on Azure Databricks lets you:

  • Query and compare Langfuse-instrumented calls together with traces from other frameworks in a single place.
  • Use Azure Databricks SQL to analyze trace data at scale.
  • Apply Unity Catalog governance such as access controls and lineage to all your traces.

Requirements

  • A Unity Catalog-enabled workspace.
  • Permissions to create catalogs and schemas in Unity Catalog.
  • A Databricks SQL warehouse with CAN USE permission for viewing traces.
  • A workspace in a supported region. See Features with limited regional availability.
  • An MLflow experiment with a UC trace location. For setup details, see Requirements.
  • The following Unity Catalog permissions on the catalog and schema used to store traces:
    • USE_CATALOG and USE_SCHEMA on the catalog and schema.
    • MODIFY and SELECT on the <table_prefix>_otel_* tables. See Grant permissions.
    • CREATE TABLE on the schema, so that the setup step can create the trace tables for the new experiment.

Step 1: Install packages

Install the required packages in your Azure Databricks notebook:

%pip install "langfuse>=3.14.5" "mlflow[databricks]>=3.14.0" opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp-proto-http
%restart_python

Step 2: Disable Langfuse trace collection

Langfuse initializes its SDK from the LANGFUSE_HOST, LANGFUSE_PUBLIC_KEY, and LANGFUSE_SECRET_KEY environment variables. Set them to dummy values so spans go only to the Azure Databricks exporter added in step 5.

import os

os.environ["LANGFUSE_HOST"] = "localhost"
os.environ["LANGFUSE_PUBLIC_KEY"] = ""
os.environ["LANGFUSE_SECRET_KEY"] = ""

Note

Alternatively, set LANGFUSE_TRACING_ENABLED=False to disable Langfuse's built-in trace collection.

Step 3: Configure the Azure Databricks connection

Retrieve your workspace host URL and API token. In a Azure Databricks notebook, get them from the notebook context:

# If running outside a Databricks notebook, set DATABRICKS_HOST and DATABRICKS_TOKEN environment variables manually.
context = dbutils.notebook.entry_point.getDbutils().notebook().getContext()
DATABRICKS_HOST = context.apiUrl().get().rstrip("/")
DATABRICKS_TOKEN = context.apiToken().get()

Bind a Unity Catalog catalog, schema, and table prefix to an MLflow experiment. This tells Azure Databricks where to store incoming traces.

import mlflow
from mlflow.entities.trace_location import UnityCatalog

experiment = mlflow.set_experiment(
    experiment_name="<MLFLOW_EXPERIMENT_NAME>",
    trace_location=UnityCatalog(
        catalog_name="<UC_CATALOG_NAME>",
        schema_name="<UC_SCHEMA_NAME>",
        table_prefix="<UC_TABLE_PREFIX>",
    ),
)

For full setup details, see Create an experiment with a Unity Catalog trace location.

Step 5: Add the Azure Databricks OTLP exporter

Retrieve the TracerProvider that Langfuse registers as the global OTel provider, then attach a BatchSpanProcessor that points to the Azure Databricks OTLP endpoint. The X-Databricks-UC-Table-Name header routes incoming spans to the Unity Catalog table defined by the trace location.

from langfuse import get_client
from opentelemetry import trace as otel_trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace.export import BatchSpanProcessor

# Initialize the Langfuse client. Langfuse registers its own TracerProvider as the global OTel TracerProvider.
langfuse = get_client()

# Retrieve the global TracerProvider to attach an additional span processor.
provider = otel_trace.get_tracer_provider()

databricks_exporter = OTLPSpanExporter(
    endpoint=f"{DATABRICKS_HOST}/api/2.0/otel/v1/traces",
    headers={
        "content-type": "application/x-protobuf",
        "Authorization": f"Bearer {DATABRICKS_TOKEN}",
        "X-Databricks-UC-Table-Name": experiment.trace_location.full_otel_spans_table_name,
    },
)

# Because the Langfuse env vars are set to dummy values, this processor is the only
# active exporter, so all spans go exclusively to Databricks.
provider.add_span_processor(BatchSpanProcessor(databricks_exporter))

Step 6: Run a traced function

Use Langfuse's @observe() decorator to instrument your agent. The decorator creates OTel spans that the exporter sends to Azure Databricks.

from langfuse import observe

@observe()
def my_llm_call(prompt: str) -> str:
    # Replace with your LLM logic (OpenAI, Anthropic, etc.)
    return f"Response to: {prompt}"

@observe()
def my_pipeline(user_input: str) -> str:
    result = my_llm_call(user_input)
    return result

my_pipeline("Hello, world!")

Step 7: View traces

Open the MLflow experiment in your Azure Databricks workspace and click the Traces tab. The trace from the my_pipeline call appears.

To search for ingested Langfuse traces by OTel span attribute values, see Search for traces by OTel span attributes.

  • Last updated on