Επεξεργασία

Κοινή χρήση μέσω


Use Azure Machine Learning studio to debug pipeline failures

After you submit a pipeline job, you can select a link to the job in your workspace in Azure Machine Learning studio. The link opens the pipeline job detail page, where you can check results and debug failed pipeline jobs. This article explains how to use the pipeline job detail page and pipeline comparison (preview) to debug machine learning pipeline failures.

Important

Items marked (preview) in this article are currently in public preview. The preview version is provided without a service level agreement, and it's not recommended for production workloads. Certain features might not be supported or might have constrained capabilities. For more information, see Supplemental Terms of Use for Microsoft Azure Previews.

Use outline to quickly find a node

On the pipeline job detail page, the Outline pane at left shows the overall structure of your pipeline job. Hover on any row and select the Locate in canvas icon to highlight that node on the canvas and open an information pane for the node on the right.

Screenshot showing outline and locate in the canvas.

In the Outline pane, you can select the Filter icon to quickly filter the view to Completed nodes only, Component only, or Data only. You can also filter the list by entering node names or component names in the Search box, or by selecting Add filter and choosing from a list of filters.

Screenshot showing quick filter and search in the Outline pane.

The left pane shows the matched nodes with more information including status, duration, and run time and date. You can sort the filtered nodes.

Screenshot of sorting search results in the Outline pane.

Check component logs and outputs

If your pipeline fails or gets stuck on a node, first view the logs.

Animated screenshot showing how to check node logs.

  1. Select the node to open the information pane at right.

  2. Select the Outputs + logs tab to view all the outputs and logs from this node.

    Screenshot of the user_logs in the node information pane.

    • The user_logs folder contains information about user code generated logs. This folder is open by default, and the std_log.txt log is selected. Your code's logs, such as print statements, appear in the std_log.txt.

    • The system_logs folder contains logs generated by Azure Machine Learning. To learn more, see View and download diagnostic logs.

    Note

    If you don't see those folders, the compute run time update might not be released to the compute cluster yet. You can look at 70_driver_log.txt in the azureml-logs folder first.

Compare pipeline jobs (preview)

You can compare different pipeline jobs to debug failure or other unexpected issues (preview). Pipeline comparison identifies the differences, such as topology, component properties, and job properties, between pipeline jobs.

You can compare successful and failed pipeline jobs to find differences that might make one pipeline job fail. You can debug a failed pipeline job by comparing it to a completed job, or debug a failed node in a pipeline by comparing it to a similar completed node.

To enable this feature in Azure Machine Learning studio, select the megaphone icon at top right to manage preview features. In the Managed preview feature panel, make sure Compare pipeline jobs to debug failures or unexpected issues is set to Enabled.

Screenshot of the preview feature toggled on.

Compare a failed pipeline job to a successful job

During iterative model development, you might clone and modify a successful baseline pipeline by changing a parameter, dataset, compute resource, or other setting. If the new pipeline fails, you can use pipeline comparison to help figure out the failure by identifying the changes from the parent pipeline.

For example, if your new pipeline failed due to an out-of-memory issue, you can use pipeline comparison to see what changes from the parent pipeline might cause memory issues.

Compare the pipeline with its parent

  1. On the failed pipeline job page, select Show lineage.

  2. Select the link in the Cloned from popup to open the parent pipeline job page in a new browser tab.

    Screenshot showing the cloned from link, with the previous step, the lineage button highlighted.

  3. On both pages, select Add to compare on the top menu bar to add both jobs to the Compare list.

    Screenshot showing the comparison list with a parent and child pipeline added.

Once you add both pipelines to the comparison list, you can select Compare details or Compare graph.

Compare graph

Compare graph shows the graph topology changes between pipelines A and B. On the canvas, nodes specific to pipeline A are marked A and highlighted in red, and nodes specific to pipeline B are marked B and highlighted in green. A description of changes appears at the tops of nodes that have differences.

You can select any node to open a Component information pane, where you can see Dataset properties or Component properties like parameters, runSettings, and outputSettings. You can choose to Show only differences and to Show differences inline.

Screenshot showing the parameter changed and the component information tab.

In this view, you can select Show compare details at upper right to open the pipeline Comparison overview, which shows the same information as the Details comparison page.

Compare details

To see overall pipeline and job metadata, properties, and differences, select Compare details in the compare list. The Details comparison page shows Pipeline properties and Job properties for both pipeline jobs.

  • Pipeline properties include pipeline parameters, compute settings, and output settings.
  • Run properties include run status, submit time and duration, and other run settings.

You can choose to Show only differences and See differences inline, or select Compare graph at upper right to open the graph topology comparison.

Screenshot of detail comparison with compare graph highlighted.

The following screenshot shows an example of using the detail comparison where the defaultCompute setting might be the reason for failure.

Screenshot showing the comparison overview of the default compute.

Compare a failed pipeline node to a similar completed node

If you only updated node properties, you can debug the node by comparing it with the same node in other jobs.

  1. Right select a failed node and select View jobs to get a list of jobs.

    Screenshot that shows a failed node with view jobs highlighted.

  2. Choose a completed job as a comparison target and open it.

  3. On both job pages, select Add to compare on the top menu bar to add both jobs to the Compare list.

  4. Once the two jobs are in the comparison list, select Compare details to show the differences.

Share debug results

To share debug results with your teammates or other stakeholders, select Share on the top menu bar. You can choose to Copy shareable link to graph or Copy pipeline job ID to share with others.

Screenshot showing the share button and the link you should copy.