Enhancement Request for AzureML Many Models Pipelines – Issue with Registered Models

Question

Hi Team,

I hope this email finds you well. My name is Raghavendra Namanio, and I am a Senior Data Scientist based in Hyderabad. I have extensive experience working with Azure ML for model building, pipeline creation, and MLOps.

Recently, while working on a project leveraging Azure Cloud, AzureML, and AzureML pipelines, I encountered an issue (or potential enhancement opportunity) related to the AzureML Many Models pipelines (referenced in (https://github.com/microsoft/solution-accelerator-many-models)). Here’s a summary of the problem:

Issue Description:

1. Training Pipelines (Parallel Execution):**
   • I created a training pipeline using the Many Models solution, partitioned by a specific column.
   • When running two training pipelines in parallel, both execute successfully. Each pipeline registers its best models under the same auto-generated name (e.g., **automl_f5c7ed0481665eac4aa5e******************), but with different versions.
2. Forecasting Pipeline Failure:**
   • During forecasting, the pipeline fails because it cannot distinguish between models from the two training runs.
   - The root cause: Even though the training pipelines are independent, their registered models share the same base name (only differing by version). This creates ambiguity when the forecasting pipeline attempts to fetch the correct model.

Suggested Enhancements:**

To resolve this, I propose the following modifications to the Many Models pipeline logic:

1. Unique Naming for Independent Runs:**
   • If a new Many Models training pipeline runs independently, it should register models with a globally unique name (e.g., incorporating the pipeline run ID ).
2. Parallel Run Support:**
   • When pipelines run in parallel, each should register models with distinct names (not just versions) to avoid conflicts during downstream forecasting.

Workaround Implemented:

For my case, I customised my pipelines to:

• Register models with the same auto-generated base name but differentiate them using tags (e.g., target variable, etc.).
• The forecasting pipeline then filters models by these tags to identify the correct version.

Could you confirm if this is a known limitation or if there are plans to address it? I’d be happy to provide additional details, examples, or even hop on a call to clarify further.

Please ignore if you are already aware of this issue or enhancement. I just wanted to update you on this.

Thank you for your time and support! Looking forward to your thoughts.

Raghava