Monitor Kubernetes Online Endpoint inference server logs

APPLIES TO: Azure CLI ml extension v2 (current) Python SDK azure-ai-ml v2 (current)

To diagnose online issues and monitor Azure Machine Learning model inference server metrics, you usually need to collect model inference server logs. In this article, learn how to collect inference server logs from Azure Kubernetes Service (AKS) and Azure Arc enabled Kubernetes clusters. The logs are collected in Log Analytics workspace, which is a part of Azure Monitor.

AKS cluster

In AKS cluster, you can use the built-in ability to collect container logs. Follow the steps to collect inference server logs in AKS:

1. Go to the AKS portal and select Logs tab

   

2. Select Configure Monitoring to enable Azure Monitor for your AKS. In the Advanced Settings section, you can specify an existing Log Analytics or create a new one for collecting logs.

   

3. After about 1 hour for it to take effect, you can query inference server logs from AKS or Log Analytics portal.

   

4. Query example:

       let starttime = ago(1d);
       ContainerLogV2
       | where TimeGenerated > starttime
       | where PodName has "blue-sklearn-mnist"
       | where ContainerName has "inference-server"
       | project TimeGenerated, PodNamespace, PodName, ContainerName, LogMessage
       | limit 100
   

Azure Arc enabled cluster

In Arc Kubernetes cluster, you can reference the Azure Monitor document to upload logs to Log Analytics from your cluster by utilizing Azure Monitor Agent