Migrate existing traces to Unity Catalog

Unity Catalog is the recommended storage location for MLflow traces. This page covers two migration paths:

  • Experiment storage → Unity Catalog: if your traces are stored in an MLflow experiment (the legacy default), migrate them to Unity Catalog for governed access, SQL queryability, and no storage cap.
  • Legacy Unity Catalog format → current Unity Catalog format: if you configured Unity Catalog trace storage using the older schema-linked format (catalog.schema), migrate to the current table-prefix format (catalog.schema.table_prefix) for faster time-range queries, richer attribute types, a dedicated annotations table, and support for multiple trace destinations per schema.

If you are starting fresh with no existing traces to migrate, see Store traces in Unity Catalog to configure a new experiment directly.

Migrate from experiment storage to Unity Catalog

This migration copies traces, spans, assessments, tags, and metadata from a source MLflow experiment to Unity Catalog Delta tables. The source experiment is not modified.

Note

The migration does not copy archived or deleted traces, dataset records, labeling sessions, runs, or non-trace entities.

Requirements

  • The prerequisites for storing traces in Unity Catalog, including the required workspace previews. See Requirements.

  • A Databricks SQL warehouse with CAN USE permission. The migration command runs on the cluster and does not use the warehouse; the warehouse is needed to view migrated traces in the UI.

  • A Azure Databricks cluster running Databricks Runtime 15.3 or above.

  • The databricks-agents Python package:

    pip install "databricks-agents>=1.10.1"

  • The following permissions:

    • Read access to the source experiment.
    • USE_CATALOG and USE_SCHEMA on the destination catalog and schema.
    • CREATE TABLE on the destination schema. The migration also creates a _migration_skipped table in the same schema if any traces are skipped.
    • MODIFY and SELECT on the destination <prefix>_otel_* tables. SELECT is required because the migration reads existing rows to skip already-migrated traces. ALL_PRIVILEGES is not sufficient — grant MODIFY and SELECT explicitly. See Grant permissions.

Step 1: Create a destination experiment

Create an MLflow experiment bound to a Unity Catalog trace location. The trace location is a three-part path (catalog.schema.table_prefix), and the migration writes to four Delta tables: <prefix>_otel_spans, <prefix>_otel_annotations, <prefix>_otel_logs, and <prefix>_otel_metrics.

import mlflow
from mlflow.entities.trace_location import UnityCatalog

experiment = mlflow.set_experiment(
    experiment_name="/Workspace/Users/<user>/<experiment_name>",
    trace_location=UnityCatalog(
        catalog_name="<destination_catalog>",
        schema_name="<destination_schema>",
        table_prefix="<table_prefix>",
    ),
)

print(f"Destination experiment ID: {experiment.experiment_id}")

Save the experiment ID — you use it in Steps 2 and 3. You can ingest a few test traces to verify Unity Catalog tracing works before proceeding. See Log traces to the Unity Catalog tables.

Step 2: Switch trace logging and stop writes

Before running the migration, redirect trace logging to the new destination and stop writes to the source experiment. This ensures no traces are lost during migration.

  1. Stop all writes to the source experiment. Verify that no notebooks, jobs, or deployed models are actively logging to it.

  2. Replace any set_experiment call that points to the source experiment:

    import mlflow

# By experiment name
mlflow.set_experiment(
    experiment_name="/Workspace/Users/<user>/<destination_experiment_name>",
)

# Or by experiment ID
mlflow.set_experiment(experiment_id="<destination_experiment_id>")

Note

Trace location can also be configured through the MLFLOW_EXPERIMENT_NAME and MLFLOW_EXPERIMENT_ID environment variables, which are used by deployed agents, containerized services, model serving endpoint configurations, and IDE or local development setups. For details, see Tracing overview and Export MLflow traces to OpenTelemetry.

Step 3: Run the migration

In a Azure Databricks notebook on the cluster, run:

from databricks.migrations.migrate_traces_to_uc import run

run(
    source_experiment_id="<source_experiment_id>",
    target_experiment_id="<destination_experiment_id>",
)

The migration is idempotent — if interrupted (for example, due to a cluster timeout), re-run the same command. It resumes from where it left off and skips already-migrated rows.

To migrate only traces created after a specific time, pass start_time_ms (epoch milliseconds):

import time
from databricks.migrations.migrate_traces_to_uc import run

one_week_ago_ms = int((time.time() - 7 * 24 * 60 * 60) * 1000)

run(
    source_experiment_id="<source_experiment_id>",
    target_experiment_id="<destination_experiment_id>",
    start_time_ms=one_week_ago_ms,  # Only migrate traces from the last 7 days
)

After the migration completes, the source experiment is not modified and can be retained as a backup. If you use production monitoring, persist a SQL warehouse ID on the destination experiment before registering scorers. See Configure a SQL warehouse for Unity Catalog traces.

Migrate from the legacy Unity Catalog format

If you configured Unity Catalog trace storage using the older schema-linked format, your traces are stored in fixed-name tables like mlflow_experiment_trace_otel_spans and mlflow_experiment_trace_otel_logs. This migration copies spans and annotations to the current table-prefix format using Spark SQL.

How to identify if you need this migration: check whether your Unity Catalog schema contains tables named mlflow_experiment_trace_otel_spans and mlflow_experiment_trace_otel_logs. If it does, your experiment uses the older schema-linked format and is a candidate for migration.

The two formats differ as follows:

  • Schema-linked (older format): the experiment's trace destination is a two-part path (catalog.schema). Trace data lives in fixed-name tables. Tags, assessments, and metadata are stored as log events in the logs table.
  • Table-prefix (current format): the trace destination is a three-part path (catalog.schema.table_prefix). Trace data lives in prefix-namespaced tables. Annotations have a dedicated table.

Requirements

Same shared prerequisites as the first migration path (UC setup, SQL warehouse, DBR 15.3+, and databricks-agents package). See the first path's requirements for the full list. Additionally:

  • On the source: USE_CATALOG, USE_SCHEMA, and SELECT on the source catalog, schema, and mlflow_experiment_trace_otel_* tables.
  • On the destination: USE_CATALOG, USE_SCHEMA, MODIFY, and SELECT on the destination catalog, schema, and <table_prefix>_otel_* tables. SELECT is required because the migration reads existing rows to skip already-migrated data.
  • CREATE TABLE on the destination schema.

Step 1: Create a destination experiment

Create an experiment linked to a Unity Catalog table-prefix location. For full setup details, see Create an experiment with a Unity Catalog trace location.

import mlflow
from mlflow.entities.trace_location import UnityCatalog

experiment = mlflow.set_experiment(
    experiment_name="/Workspace/Users/<user>/<experiment_name>",
    trace_location=UnityCatalog(
        catalog_name="<destination_catalog>",
        schema_name="<destination_schema>",
        table_prefix="<table_prefix>",
    ),
)

print(f"Experiment ID: {experiment.experiment_id}")

Save the experiment ID. Use it to configure your notebooks, jobs, or deployed models to log traces to the new destination.

Step 2: Switch trace logging and stop writes

Update your notebooks, jobs, or deployed models to log traces to the destination experiment created in Step 1.

Important

Stop all writes to the source experiment before running the migration. Any traces written to the source tables during migration might not be copied. Verify that no notebooks, jobs, or deployed models are actively logging traces to the source experiment.

If you want to do a dry run first, you can skip this step and run the migration without switching your production workloads.

Step 3: Run the migration

In a Azure Databricks notebook on the cluster, run:

from databricks.migrations.v1_to_v2 import V1ToV2SqlMigration

migration = V1ToV2SqlMigration(
    v1_source_schema="<source_catalog>.<source_schema>",
    v2_destination_prefix="<destination_catalog>.<destination_schema>.<table_prefix>",
)
migration.run()

Replace the placeholders:

  • <source_catalog>.<source_schema>: the Unity Catalog catalog and schema where your source trace tables are stored.
  • <destination_catalog>.<destination_schema>.<table_prefix>: the Unity Catalog catalog, schema, and table prefix for the destination. This must match the location configured in Step 1.

The migration is idempotent — if it fails partway through, re-run it. Already-migrated rows are skipped automatically.

After the migration completes, the source tables are not modified and can be retained as a backup. If you use production monitoring, persist a SQL warehouse ID on the destination experiment before registering scorers. See Configure a SQL warehouse for Unity Catalog traces.

Additional resources

  • Last updated on