Checklist for production-ready AI models

Creating AI models is much more than training and validating a model. This article summarizes key requirements for production-ready models.

💡Model development checklist:

• Datasets are versioned, centrally available and discoverable. Also full lineage of features and engineering is captured.
• Models trained are versioned, centrally available and discoverable.
• Unit and integration tests have been applied to the training and inference pipeline. Also static seeds applied where relevant, to ensure near identical and reproducible results.
• A consistent scoring and evaluation approach has been implemented.
• Performance and parallel processing improvements are designed, tested, monitored and partially implemented and tested. Compute environments are explored for optimum results.

The following sections describe additional elements of developing high quality production models.

ML pipelines

ML pipelines are automated workflows, which consist of ingesting training data or features, and training, validating a model prior to packaging. These pipelines are designed to be repeatable and modular to facilitate agile iteration and development.

For an in-depth look, refer to Machine Learning Pipelines

Feature management

Much of the complexity in developing ML systems is in data preparation and data normalization for feature engineering workflows. Feature management enables accessing features by name, model version, lineage, etc., in a uniform way across environments. Using feature management provides consistency to help improve developer productivity. It also promotes discoverability and reusability of features across teams.

For more information, see Feature Management

Model engineering

Model engineering consists of the process of iterating on a model, tuning hyperparameters, and preparing the preprocessing/training code for inclusion in an ML training pipeline. This process also includes details on how to integrate standard software engineering practices into model engineering, such as unit testing.

• Testing your ML code

Model validation

Model validation is the process of testing a newly trained model to ensure it works (such as, properly accepting inputs and creating expected outputs), and it performs the same or better than any previous model. Validation can include comparing metrics such as accuracy, ROC, etc.

For more information

• MLOps Model Factory Template based on SDK (v2): A machine learning (ML) model factory is a system for automatically building, training, and deploying ML models at scale. It includes various features that make it easier to create and manage large numbers of ML models. It can also automate the model building process.
• MLOps Template for Azure ML CLI (v2): An MLOps implementation for projects that are using the Azure ML CLI v2.
• MLOps Template for Azure ML SDK (v2): An MLOps implementation for projects that are using the Azure ML SDK v2.
• Azure MLOps (v2) solution accelerator: This project can be a starting point for MLOps implementation in Azure ML CLI or SDK v2.