Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
MLflow tracing is built on OpenTelemetry (OTel). Because every trace MLflow captures is a set of OTel spans, you can export those spans outbound to any OpenTelemetry-compatible observability platform — such as Datadog, Grafana, or Splunk — over the standard OTLP protocol. You can send traces exclusively to an external collector, or use dual export to send them to both Azure Databricks MLflow and your existing observability stack at the same time.
Export MLflow traces to an external OTel collector
MLflow can export traces outbound to any OpenTelemetry-compatible observability platform. Three modes are supported.
OTel-only export
To send traces exclusively to an external OTel collector (bypassing Azure Databricks MLflow storage), set OTEL_EXPORTER_OTLP_TRACES_ENDPOINT before starting any trace:
import os
import mlflow
os.environ["OTEL_EXPORTER_OTLP_TRACES_ENDPOINT"] = "http://localhost:4317/v1/traces"
os.environ["OTEL_SERVICE_NAME"] = "<your-service-name>"
# Trace is exported ONLY to the OTel collector at http://localhost:4317/v1/traces
with mlflow.start_span(name="foo") as span:
span.set_inputs({"a": 1})
span.set_outputs({"b": 2})
Dual export (MLflow + OTel)
To export traces to both Azure Databricks MLflow and an external OTel collector simultaneously, set MLFLOW_ENABLE_DUAL_EXPORT:
import os
import mlflow
os.environ["MLFLOW_ENABLE_DUAL_EXPORT"] = "true"
os.environ["OTEL_EXPORTER_OTLP_TRACES_ENDPOINT"] = "http://localhost:4317/v1/traces"
os.environ["OTEL_SERVICE_NAME"] = "my-ml-service"
mlflow.set_tracking_uri("databricks")
# Traces are exported to BOTH MLflow and the OTel collector
with mlflow.start_span(name="dual_export_example") as span:
span.set_inputs({"model": "gpt-4", "prompt": "Hello world"})
span.set_outputs({"response": "Generated response"})
Metrics export
MLflow exports OTel metrics when a metrics endpoint is configured. For the complete list of exported metrics, see the MLflow documentation.
import os
os.environ["OTEL_METRICS_EXPORTER"] = "otlp"
os.environ["OTEL_EXPORTER_OTLP_METRICS_ENDPOINT"] = "http://localhost:4317"
# Optional: configure export interval in milliseconds
os.environ["OTEL_METRIC_EXPORT_INTERVAL"] = "60000"
Supported platforms
MLflow uses the standard OTLP exporter and supports all OTLP exporter configurations. The following platforms have published OTel setup guides:
| Platform | OpenTelemetry documentation |
|---|---|
| Datadog | OpenTelemetry Guide |
| New Relic | OpenTelemetry APM Monitoring |
| SigNoz | OpenTelemetry Python Instrumentation |
| Splunk | Get Data In |
| Grafana | Send Data via OTLP |
| ServiceNow (Lightstep) | Collector Documentation |
To use HTTP protocol instead of the default gRPC or to set custom headers:
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="http://localhost:4317/v1/traces"
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL="http/protobuf"
export OTEL_EXPORTER_OTLP_TRACES_HEADERS="api_key=12345"
Additional resources
- Export Langfuse traces to Azure Databricks — Route Langfuse traces to Azure Databricks over the OTLP endpoint.
- Custom OpenTelemetry instrumentation — Set OTel span attributes for a custom-instrumented agent and search ingested traces.
- Store OpenTelemetry traces in Unity Catalog — Store and manage traces in Unity Catalog tables.
- View traces in the Databricks MLflow UI — Search and filter traces in the MLflow UI.
- Observe and find issues — Query trace data at scale using Databricks SQL.
- Programmatic access to traces — Search and filter traces programmatically with the MLflow SDK.
- Enrich traces: tags, context, and feedback — Attach searchable tags to traces.
Next step: Observe and find issues