AKS - Difference between on box log and logs on portal

In Azure Kubernetes Service, the logs available in the Azure portal give you deep visibility into the managed control plane components that Microsoft runs behind the scenes. Enabling these categories routes telemetry directly to your Log Analytics workspace. The Kubernetes API Server logs record every single API request, authentication attempt, and resource modification in the cluster. The Kubernetes Controller Manager logs track the background control loops that regulate cluster state, such as namespace lifecycle events and pod replication. The Kubernetes Scheduler logs reveal the exact decision-making process behind pod assignments, helping you diagnose why a pod is stuck in a pending state.

The remaining portal log streams focus on infrastructure scaling and cloud ecosystem integration. The Kubernetes Cluster Autoscaler logs detail the automated decisions to scale the node pools up or down based on pending pod resource requests. The Kubernetes Cloud Controller Manager logs capture the direct interactions between the Kubernetes cluster and the underlying Azure fabric, such as the provisioning of Azure Load Balancers or the attachment of managed Azure Disks. The Kubernetes Node Auto Provisioning logs track dynamic, just-in-time node allocation events, detailing how and why specific virtual machine sizes or architectures were selected to fulfill un-schedulable workloads.

These control plane logs are different from those stored locally on the AKS worker nodes themselves, which hold host-level runtime data. On the physical or virtual worker nodes, you can access the local kubelet service logs, containerd container runtime events, the waagent host bootstrapping files, and raw operating system messages. AFAIK, considering that Microsoft isolates the master control plane infrastructure from your data plane worker nodes, components like the API Server or the Cluster Autoscaler do not write logs to the worker node file systems. This means that if a worker node crashes entirely, these portal logs remain safely stored in the cloud for historical analysis.

However, generally, you might not to pull logs from both locations simultaneously during standard cluster administration. The recommended workflow is to use the centralized portal logs for all application debugging, security auditing, scheduling failures, and scaling analysis. Direct node-level log retrieval is a secondary troubleshooting step that you only need to perform if the worker node enters a NotReady state, if the Azure Monitor logging agent stops transmitting data, or if you suspect a low-level host kernel or virtual networking fault.

If the above response helps answer your question, remember to "Accept Answer" so that others in the community facing similar issues can easily find the solution. Your contribution is highly appreciated.

hth

Marcin