Migrate logging from SDK v1 to SDK v2

Azure Machine Learning uses MLflow Tracking for metric logging and artifact storage for your experiments, whether you created the experiments via the Azure Machine Learning Python SDK, the Azure Machine Learning CLI, or Azure Machine Learning studio. We recommend using MLflow for tracking experiments.

If you're migrating from SDK v1 to SDK v2, use the information in this section to understand the MLflow equivalents of SDK v1 logging APIs.

Why MLflow?

MLflow, with over 13 million monthly downloads, has become the standard platform for end-to-end MLOps, enabling teams of all sizes to track, share, package and deploy any model for batch or real-time inference. Azure Machine Learning integrates with MLflow, which enables your training code to achieve true portability and seamless integration with other platforms since it doesn't hold any Azure Machine Learning specific instructions.

Prepare for migrating to MLflow

To use MLflow tracking, you need to install Mlflow SDK package mlflow and Azure Machine Learning plug-in for MLflow azureml-mlflow. All Azure Machine Learning environments have these packages already available for you but you need to include them if creating your own environment.

pip install mlflow azureml-mlflow

Connect to your workspace

Azure Machine Learning allows users to perform tracking in training jobs running on your workspace or running remotely (tracking experiments running outside Azure Machine Learning). If performing remote tracking, you need to indicate the workspace you want to connect MLflow to.

Azure Machine Learning compute

You are already connected to your workspace when running on Azure Machine Learning compute.

Remote compute

Configure tracking URI

1. Get the tracking URI for your workspace:

   Azure CLI

   APPLIES TO: Azure CLI ml extension v2 (current)

   1. Login and configure your workspace:

      az account set --subscription <subscription>
      az configure --defaults workspace=<workspace> group=<resource-group> location=<location> 
      

   2. You can get the tracking URI using the az ml workspace command:

      az ml workspace show --query mlflow_tracking_uri
      

   Python

   APPLIES TO: Python SDK azure-ai-ml v2 (current)

   You can get the Azure ML MLflow tracking URI using the Azure Machine Learning SDK v2 for Python. Ensure you have the library azure-ai-ml installed in the compute you are using. The following sample gets the unique MLFLow tracking URI associated with your workspace.

   1. Login into your workspace using the MLClient. The easier way to do that is by using the workspace config file:

      from azure.ai.ml import MLClient
      from azure.identity import DefaultAzureCredential
      
      ml_client = MLClient.from_config(credential=DefaultAzureCredential())
      

      Tip

      You can download the workspace configuration file by:

      1. Navigate to Azure ML studio
      2. Click on the upper-right corner of the page -> Download config file.
      3. Save the file config.json in the same directory where you are working on.

   2. Alternatively, you can use the subscription ID, resource group name and workspace name to get it:

      from azure.ai.ml import MLClient
      from azure.identity import DefaultAzureCredential
      
      #Enter details of your AzureML workspace
      subscription_id = '<SUBSCRIPTION_ID>'
      resource_group = '<RESOURCE_GROUP>'
      workspace_name = '<WORKSPACE_NAME>'
      
      ml_client = MLClient(credential=DefaultAzureCredential(),
                              subscription_id=subscription_id, 
                              resource_group_name=resource_group,
                              workspace_name=workspace_name)
      

      Important

      DefaultAzureCredential will try to pull the credentials from the available context. If you want to specify credentials in a different way, for instance using the web browser in an interactive way, you can use InteractiveBrowserCredential or any other method available in azure.identity package.

   3. Get the Azure Machine Learning Tracking URI:

      mlflow_tracking_uri = ml_client.workspaces.get(ml_client.workspace_name).mlflow_tracking_uri
      

   Studio

   Use the Azure Machine Learning portal to get the tracking URI:

   1. Open the Azure Machine Learning studio portal and log in using your credentials.

   2. In the upper right corner, click on the name of your workspace to show the Directory + Subscription + Workspace blade.

   3. Click on View all properties in Azure Portal.

   4. On the Essentials section, you will find the property MLflow tracking URI.

   Manually

   The Azure Machine Learning Tracking URI can be constructed using the subscription ID, region of where the resource is deployed, resource group name and workspace name. The following code sample shows how:

   Warning

   If you are working in a private link-enabled workspace, the MLflow endpoint will also use a private link to communicate with Azure Machine Learning. As a consequence, the tracking URI will look different as proposed here. You need to get the tracking URI using the Azure ML SDK or CLI v2 on those cases.

   region = "<LOCATION>"
   subscription_id = '<SUBSCRIPTION_ID>'
   resource_group = '<RESOURCE_GROUP>'
   workspace_name = '<AML_WORKSPACE_NAME>'
   
   mlflow_tracking_uri = f"azureml://{region}.api.azureml.ms/mlflow/v1.0/subscriptions/{subscription_id}/resourceGroups/{resource_group}/providers/Microsoft.MachineLearningServices/workspaces/{workspace_name}"
   

2. Configuring the tracking URI:

   Using MLflow SDK

   Then the method set_tracking_uri() points the MLflow tracking URI to that URI.

   import mlflow
   
   mlflow.set_tracking_uri(mlflow_tracking_uri)
   

   Using environment variables

   You can set the MLflow environment variables MLFLOW_TRACKING_URI in your compute to make any interaction with MLflow in that compute to point by default to Azure Machine Learning.

   MLFLOW_TRACKING_URI=$(az ml workspace show --query mlflow_tracking_uri | sed 's/"//g') 
   

   Tip

   When working on shared environments, like an Azure Databricks cluster, Azure Synapse Analytics cluster, or similar, it is useful to set the environment variable MLFLOW_TRACKING_URI at the cluster level to automatically configure the MLflow tracking URI to point to Azure Machine Learning for all the sessions running in the cluster rather than to do it on a per-session basis.

Configure authentication

Once the tracking is configured, you also need to configure how the authentication needs to happen to the associated workspace. By default, the Azure Machine Learning plugin for MLflow performs interactive authentication by opening the default browser to prompt for credentials. Refer to Configure MLflow for Azure Machine Learning: Configure authentication for more ways to configure authentication for MLflow in Azure Machine Learning workspaces.

For interactive jobs where there's a user connected to the session, you can rely on Interactive Authentication and hence no further action is required.

Warning

Interactive browser authentication blocks code execution when it prompts for credentials. This approach isn't suitable for authentication in unattended environments like training jobs. We recommend that you configure a different authentication mode.

For those scenarios where unattended execution is required, you have to configure a service principal to communicate with Azure Machine Learning.

MLflow SDK

import os

os.environ["AZURE_TENANT_ID"] = "<AZURE_TENANT_ID>"
os.environ["AZURE_CLIENT_ID"] = "<AZURE_CLIENT_ID>"
os.environ["AZURE_CLIENT_SECRET"] = "<AZURE_CLIENT_SECRET>"

Using environment variables

export AZURE_TENANT_ID="<AZURE_TENANT_ID>"
export AZURE_CLIENT_ID="<AZURE_CLIENT_ID>"
export AZURE_CLIENT_SECRET="<AZURE_CLIENT_SECRET>"

Tip

When working on shared environments, we recommend that you configure these environment variables at the compute. As a best practice, manage them as secrets in an instance of Azure Key Vault.

For instance, in Azure Databricks you can use secrets in environment variables as follows in the cluster configuration: AZURE_CLIENT_SECRET={{secrets/<scope-name>/<secret-name>}}. For more information about implementing this approach in Azure Databricks, see Reference a secret in an environment variable or refer to documentation for your platform.

Experiments and runs

SDK v1

from azureml.core import Experiment

# create an Azure Machine Learning experiment and start a run
experiment = Experiment(ws, "create-experiment-sdk-v1")
azureml_run = experiment.start_logging()

SDK v2 with MLflow

# Set the MLflow experiment and start a run
mlflow.set_experiment("logging-with-mlflow")
mlflow_run = mlflow.start_run()

Logging API comparison

Log an integer or float metric

SDK v1

azureml_run.log("sample_int_metric", 1)

SDK v2 with MLflow

mlflow.log_metric("sample_int_metric", 1)

Log a boolean metric

SDK v1

azureml_run.log("sample_boolean_metric", True)

SDK v2 with MLflow

mlflow.log_metric("sample_boolean_metric", 1)

Log a string metric

SDK v1

azureml_run.log("sample_string_metric", "a_metric")

SDK v2 with MLflow

mlflow.log_text("sample_string_text", "string.txt")

• The string is logged as an artifact, not as a metric. In Azure Machine Learning studio, the value is displayed in the Outputs + logs tab.

Log an image to a PNG or JPEG file

SDK v1

azureml_run.log_image("sample_image", path="Azure.png")

SDK v2 with MLflow

mlflow.log_artifact("Azure.png")

The image is logged as an artifact and it appears in the Images tab in Azure Machine Learning Studio.

Log a matplotlib.pyplot

SDK v1

import matplotlib.pyplot as plt

plt.plot([1, 2, 3])
azureml_run.log_image("sample_pyplot", plot=plt)

SDK v2 with MLflow

import matplotlib.pyplot as plt

plt.plot([1, 2, 3])
fig, ax = plt.subplots()
ax.plot([0, 1], [2, 3])
mlflow.log_figure(fig, "sample_pyplot.png")

• The image is logged as an artifact and it appears in the Images tab in Azure Machine Learning Studio.

Log a list of metrics

SDK v1

list_to_log = [1, 2, 3, 2, 1, 2, 3, 2, 1]
azureml_run.log_list('sample_list', list_to_log)

SDK v2 with MLflow

list_to_log = [1, 2, 3, 2, 1, 2, 3, 2, 1]
from mlflow.entities import Metric
from mlflow.tracking import MlflowClient
import time

metrics = [Metric(key="sample_list", value=val, timestamp=int(time.time() * 1000), step=0) for val in list_to_log]
MlflowClient().log_batch(mlflow_run.info.run_id, metrics=metrics)

• Metrics appear in the metrics tab in Azure Machine Learning studio.
• Text values are not supported.

Log a row of metrics

SDK v1

azureml_run.log_row("sample_table", col1=5, col2=10)

SDK v2 with MLflow

metrics = {"sample_table.col1": 5, "sample_table.col2": 10}
mlflow.log_metrics(metrics)

• Metrics do not render as a table in Azure Machine Learning studio.
• Text values are not supported.
• Logged as an artifact, not as a metric.

Log a table

SDK v1

table = {
"col1" : [1, 2, 3],
"col2" : [4, 5, 6]
}
azureml_run.log_table("table", table)

SDK v2 with MLflow

# Add a metric for each column prefixed by metric name. Similar to log_row
row1 = {"table.col1": 5, "table.col2": 10}
# To be done for each row in the table
mlflow.log_metrics(row1)

# Using mlflow.log_artifact
import json

with open("table.json", 'w') as f:
json.dump(table, f)
mlflow.log_artifact("table.json")

• Logs metrics for each column.
• Metrics do not render as a table in Azure Machine Learning studio.
• Text values are not supported.
• Logged as an artifact, not as a metric.

Log an accuracy table

SDK v1

ACCURACY_TABLE = '{"schema_type": "accuracy_table", "schema_version": "v1", "data": {"probability_tables": ' +\
        '[[[114311, 385689, 0, 0], [0, 0, 385689, 114311]], [[67998, 432002, 0, 0], [0, 0, ' + \
        '432002, 67998]]], "percentile_tables": [[[114311, 385689, 0, 0], [1, 0, 385689, ' + \
        '114310]], [[67998, 432002, 0, 0], [1, 0, 432002, 67997]]], "class_labels": ["0", "1"], ' + \
        '"probability_thresholds": [0.52], "percentile_thresholds": [0.09]}}'

azureml_run.log_accuracy_table('v1_accuracy_table', ACCURACY_TABLE)

SDK v2 with MLflow

ACCURACY_TABLE = '{"schema_type": "accuracy_table", "schema_version": "v1", "data": {"probability_tables": ' +\
        '[[[114311, 385689, 0, 0], [0, 0, 385689, 114311]], [[67998, 432002, 0, 0], [0, 0, ' + \
        '432002, 67998]]], "percentile_tables": [[[114311, 385689, 0, 0], [1, 0, 385689, ' + \
        '114310]], [[67998, 432002, 0, 0], [1, 0, 432002, 67997]]], "class_labels": ["0", "1"], ' + \
        '"probability_thresholds": [0.52], "percentile_thresholds": [0.09]}}'

mlflow.log_dict(ACCURACY_TABLE, 'mlflow_accuracy_table.json')

• Metrics do not render as an accuracy table in Azure Machine Learning studio.
• Logged as an artifact, not as a metric.
• The mlflow.log_dict method is experimental.

Log a confusion matrix

SDK v1

CONF_MATRIX = '{"schema_type": "confusion_matrix", "schema_version": "v1", "data": {"class_labels": ' + \
    '["0", "1", "2", "3"], "matrix": [[3, 0, 1, 0], [0, 1, 0, 1], [0, 0, 1, 0], [0, 0, 0, 1]]}}'

azureml_run.log_confusion_matrix('v1_confusion_matrix', json.loads(CONF_MATRIX))

SDK v2 with MLflow

CONF_MATRIX = '{"schema_type": "confusion_matrix", "schema_version": "v1", "data": {"class_labels": ' + \
    '["0", "1", "2", "3"], "matrix": [[3, 0, 1, 0], [0, 1, 0, 1], [0, 0, 1, 0], [0, 0, 0, 1]]}}'

mlflow.log_dict(CONF_MATRIX, 'mlflow_confusion_matrix.json')

• Metrics do not render as a confusion matrix in Azure Machine Learning studio.
• Logged as an artifact, not as a metric.
• The mlflow.log_dict method is experimental.

Log predictions

SDK v1

PREDICTIONS = '{"schema_type": "predictions", "schema_version": "v1", "data": {"bin_averages": [0.25,' + \
    ' 0.75], "bin_errors": [0.013, 0.042], "bin_counts": [56, 34], "bin_edges": [0.0, 0.5, 1.0]}}'

azureml_run.log_predictions('test_predictions', json.loads(PREDICTIONS))

SDK v2 with MLflow

PREDICTIONS = '{"schema_type": "predictions", "schema_version": "v1", "data": {"bin_averages": [0.25,' + \
    ' 0.75], "bin_errors": [0.013, 0.042], "bin_counts": [56, 34], "bin_edges": [0.0, 0.5, 1.0]}}'

mlflow.log_dict(PREDICTIONS, 'mlflow_predictions.json')

• Metrics do not render as a confusion matrix in Azure Machine Learning studio.
• Logged as an artifact, not as a metric.
• The mlflow.log_dict method is experimental.

Log residuals

SDK v1

RESIDUALS = '{"schema_type": "residuals", "schema_version": "v1", "data": {"bin_edges": [100, 200, 300], ' + \
'"bin_counts": [0.88, 20, 30, 50.99]}}'

azureml_run.log_residuals('test_residuals', json.loads(RESIDUALS))

SDK v2 with MLflow

RESIDUALS = '{"schema_type": "residuals", "schema_version": "v1", "data": {"bin_edges": [100, 200, 300], ' + \
'"bin_counts": [0.88, 20, 30, 50.99]}}'

mlflow.log_dict(RESIDUALS, 'mlflow_residuals.json')

• Metrics do not render as a confusion matrix in Azure Machine Learning studio.
• Logged as an artifact, not as a metric.
• The mlflow.log_dict method is experimental.

View run info and data

You can access run information using the properties data and info of the MLflow run (mlflow.entities.Run) object.

Tip

Experiments and runs tracking information in Azure Machine Learning can be queried using MLflow, which provides a comprehensive search API to query and search for experiments and runs easily, and quickly compare results. For more information about all the capabilities in MLflow in this dimension, see Query & compare experiments and runs with MLflow

The following example shows how to retrieve a finished run:

from mlflow.tracking import MlflowClient

# Use MlFlow to retrieve the run that was just completed
client = MlflowClient()
finished_mlflow_run = MlflowClient().get_run("<RUN_ID>")

The following example shows how to view the metrics, tags, and params:

metrics = finished_mlflow_run.data.metrics
tags = finished_mlflow_run.data.tags
params = finished_mlflow_run.data.params

Note

The metrics will only have the most recently logged value for a given metric. For example, if you log in order a value of 1, then 2, 3, and finally 4 to a metric named sample_metric, only 4 will be present in the metrics dictionary. To get all metrics logged for a specific named metric, use MlFlowClient.get_metric_history:

with mlflow.start_run() as multiple_metrics_run:
    mlflow.log_metric("sample_metric", 1)
    mlflow.log_metric("sample_metric", 2)
    mlflow.log_metric("sample_metric", 3)
    mlflow.log_metric("sample_metric", 4)

print(client.get_run(multiple_metrics_run.info.run_id).data.metrics)
print(client.get_metric_history(multiple_metrics_run.info.run_id, "sample_metric"))

For more information, see the MlFlowClient reference.

The info field provides general information about the run, such as start time, run ID, experiment ID, etc.:

run_start_time = finished_mlflow_run.info.start_time
run_experiment_id = finished_mlflow_run.info.experiment_id
run_id = finished_mlflow_run.info.run_id

View run artifacts

To view the artifacts of a run, use MlFlowClient.list_artifacts:

client.list_artifacts(finished_mlflow_run.info.run_id)

To download an artifact, use mlflow.artifacts.download_artifacts:

mlflow.artifacts.download_artifacts(run_id=finished_mlflow_run.info.run_id, artifact_path="Azure.png")

Next steps

• Track ML experiments and models with MLflow.
• Log metrics, parameters and files with MLflow.
• Logging MLflow models.
• Query & compare experiments and runs with MLflow.
• Manage models registries in Azure Machine Learning with MLflow.