Check performance for batch endpoints

Question

My case is I want to watch and get all the data for my batch endpoints in machine learning service. I want to see all the data so that I can track my models in a time consistent way. Alert is needs in this case, is that doable?

Answer 1

@Setsu Pingh

Thanks for reaching out to us, I think the new feature should be a good choice for you -

Monitor performance of models deployed to production

https://learn.microsoft.com/en-us/azure/machine-learning/how-to-monitor-model-performance?view=azureml-api-2&tabs=azure-studio

As the description says perform out-of box and advanced monitoring setup for models that are deployed to Azure Machine Learning online endpoints. You also learn to set up monitoring for models that are deployed outside Azure Machine Learning or deployed to Azure Machine Learning batch endpoints.

Once a machine learning model is in production, it's important to critically evaluate the inherent risks associated with it and identify blind spots that could adversely affect your business. Azure Machine Learning's model monitoring continuously tracks the performance of models in production by providing a broad view of monitoring signals and alerting you to potential issues. You can do it with Studio, SDK or Azure CLI, but there are some requirements, please take a look -

• Azure role-based access controls (Azure RBAC) are used to grant access to operations in Azure Machine Learning. To perform the steps in this article, your user account must be assigned the owner or contributor role for the Azure Machine Learning workspace, or a custom role allowing Microsoft.MachineLearningServices/workspaces/onlineEndpoints/*. For more information, see Manage access to an Azure Machine Learning workspace.
• For monitoring a model that is deployed to an Azure Machine Learning online endpoint (managed online endpoint or Kubernetes online endpoint), be sure to:
  • Have a model already deployed to an Azure Machine Learning online endpoint. Both managed online endpoint and Kubernetes online endpoint are supported. If you don't have a model deployed to an Azure Machine Learning online endpoint, see Deploy and score a machine learning model by using an online endpoint.
  • Enable data collection for your model deployment. You can enable data collection during the deployment step for Azure Machine Learning online endpoints. For more information, see Collect production data from models deployed to a real-time endpoint.
• For monitoring a model that is deployed to an Azure Machine Learning batch endpoint or deployed outside of Azure Machine Learning, be sure to:
  • Have a means to collect production data and register it as an Azure Machine Learning data asset.
  • Update the registered data asset continuously for model monitoring.
  • (Recommended) Register the model in an Azure Machine Learning workspace, for lineage tracking.

The document for this feature is here- https://learn.microsoft.com/en-us/azure/machine-learning/how-to-monitor-model-performance?view=azureml-api-2&tabs=azure-studio

I hope this helps.

Regards,

Yutong

-Please kindly accept the answer and vote 'Yes' if you feel helpful to support the community, thanks a lot.