Edit

Deploy Periscope to an Azure Kubernetes Service (AKS) cluster

AKS Periscope is an open-source diagnostic tool that collects logs and diagnostic information from Azure Kubernetes Service (AKS) nodes and pods. To deploy Periscope and collect diagnostics, run az aks kollect and specify your cluster and an Azure storage account. You can also deploy Periscope using the AKS extension for Visual Studio Code, Kustomize, or Azure Copilot.

Periscope collects the following information by default:

  • Container logs from the kube-system namespace.
  • Kubelet and container runtime system service logs.
  • Outbound connectivity checks for the internet, API server, tunnel, Azure Container Registry, and Microsoft Container Registry.
  • Node iptables data, provisioning logs, cloud-init logs, and DNS settings.
  • Descriptions of pods, services, and deployments in the kube-system namespace.
  • Kubelet command arguments and node and pod resource usage.

Prerequisites

  • You need an existing AKS cluster with Linux or Windows nodes. Periscope supports both operating systems, but the collected information differs. For more information, see Windows and Linux functional differences.
  • To deploy Periscope using the Azure CLI or Kustomize, install the Azure CLI and sign in using az login.
  • To deploy Periscope using the Azure CLI, Visual Studio Code, or Kustomize, install kubectl version 1.21 or later.
  • Create or identify an Azure storage account for the collected diagnostics.
  • Verify that your cluster can pull images from Microsoft Container Registry (MCR).

Important

Periscope runs on your agent pool nodes and collects virtual machine (VM) and container-level data. Before you deploy Periscope, confirm that the cluster owner understands and consents to the collection and sharing of this information. For more information, see Microsoft Support diagnostic information collection.

Deploy Periscope using the Azure CLI

The az aks kollect command deploys Periscope, collects the diagnostic information, and uploads the results to Azure Blob Storage.

  1. Install or update the aks-preview extension.

    az extension add --name aks-preview --upgrade

  2. Set environment variables for your AKS cluster.

    RESOURCE_GROUP=<resource-group-name>
AKS_CLUSTER=<aks-cluster-name>

  3. Deploy Periscope and collect diagnostics using one of the following storage account options.

    • If the AKS diagnostic settings already specify a storage account, run the following command:

      az aks kollect \
    --resource-group $RESOURCE_GROUP \
    --name $AKS_CLUSTER

    • To use a storage account that you own, specify its resource ID:

      STORAGE_ACCOUNT_ID=<storage-account-resource-id>

az aks kollect \
    --resource-group $RESOURCE_GROUP \
    --name $AKS_CLUSTER \
    --storage-account $STORAGE_ACCOUNT_ID

    • To use a storage account name and a shared access signature (SAS) token, specify a SAS token that has write permission:

      STORAGE_ACCOUNT=<storage-account-name>
SAS_TOKEN=<sas-token>

az aks kollect \
    --resource-group $RESOURCE_GROUP \
    --name $AKS_CLUSTER \
    --storage-account $STORAGE_ACCOUNT \
    --sas-token "$SAS_TOKEN"

    Caution

    A SAS token grants access to your storage account. Protect it like a password, don't commit it to source control, and revoke or rotate it when you no longer need it.

When collection finishes, Periscope stores the results in a blob container named after the cluster API server fully qualified domain name (FQDN). The container also includes a ZIP file that you can download for analysis or share with Microsoft Support.

Customize the collected diagnostics

By default, Periscope collects logs and Kubernetes objects from the kube-system namespace. Use the optional az aks kollect parameters to target other resources.

  • Collect all container logs in a namespace or a specific container:

    az aks kollect \
    --resource-group $RESOURCE_GROUP \
    --name $AKS_CLUSTER \
    --container-logs "<namespace> <namespace>/<container-name>"

  • Describe all objects of a resource type in a namespace or a specific object:

    az aks kollect \
    --resource-group $RESOURCE_GROUP \
    --name $AKS_CLUSTER \
    --kube-objects "<namespace>/<resource-type> <namespace>/<resource-type>/<resource-name>"

  • Collect specific Linux node log files:

    az aks kollect \
    --resource-group $RESOURCE_GROUP \
    --name $AKS_CLUSTER \
    --node-logs "/var/log/azure-vnet.log /var/log/azure-vnet-ipam.log"

  • Collect specific Windows node log files:

    az aks kollect \
    --resource-group $RESOURCE_GROUP \
    --name $AKS_CLUSTER \
    --node-logs-windows 'C:\AzureData\CustomDataSetupScript.log'

You can combine these parameters with any storage account option from Deploy Periscope using the Azure CLI. For the complete command reference, see az aks kollect.

Deploy Periscope using Visual Studio Code

The AKS extension for Visual Studio Code provides a graphical workflow for deploying Periscope and downloading the results.

  1. Install the Azure Kubernetes Service extension for Visual Studio Code.
  2. Sign in to Azure in Visual Studio Code.
  3. Confirm that the AKS cluster diagnostic settings specify a storage account. If you need to configure one, complete the following steps:
    1. In the AKS extension, right-click your cluster and select Show In Azure Portal.
    2. In the Azure portal, select Monitoring > Diagnostic settings > Add diagnostic setting.
    3. Enter a name for the diagnostic setting, select Archive to a storage account, and select a storage account.
    4. Select the logs that you want to enable, and then select Save.
  4. In the AKS extension, right-click your cluster and select Run AKS Periscope.
  5. If the cluster has more than one storage account in its diagnostic settings, select the account where you want to store the results.
  6. When collection finishes, select Generate Link to create a downloadable link or a shareable link that expires after seven days. If the results don't include every node, wait for the remaining uploads and then select Generate Link again.

For more information, see AKS tools and diagnostics in Visual Studio Code.

Deploy Periscope using Kustomize

Use Kustomize when you need to inspect or override the Kubernetes resources, choose a specific Periscope release, or customize the collectors. This method requires you to manage the storage credentials and manifests directly.

  1. Create a directory for the Periscope configuration, and then create a kustomization.yaml file in that directory.

  2. Create an account SAS with the following settings. Use the shortest practical expiration time.

    • Service: Blob (ss=b).
    • Resource types: Service, container, and object (srt=sco).
    • Permissions: Read, list, add, create, and write (sp=rlacw).
    • Allowed protocol: HTTPS only.

  3. Add the Periscope base release, image tags, storage destination, SAS key, and run ID to the file. Replace every placeholder with your values. The SAS key must include the leading question mark (?).

    apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - https://github.com/Azure/aks-periscope//deployment/base?ref=<release-tag>

images:
  - name: periscope-linux
    newName: mcr.microsoft.com/aks/periscope
    newTag: <image-tag>
  - name: periscope-windows
    newName: mcr.microsoft.com/aks/periscope
    newTag: <image-tag>

secretGenerator:
  - name: azureblob-secret
    behavior: replace
    literals:
      - AZURE_BLOB_ACCOUNT_NAME=<storage-account-name>
      - AZURE_BLOB_CONTAINER_NAME=<container-name>
      - AZURE_BLOB_SAS_KEY=?<sas-token>

configMapGenerator:
  - name: diagnostic-config
    behavior: merge
    literals:
      - DIAGNOSTIC_RUN_ID=<YYYY-MM-DDThh-mm-ssZ>

    Caution

    The kustomization.yaml file contains the SAS token in plaintext. Don't commit the file to source control. Protect the SAS like an account key, and revoke or rotate it when you no longer need it.

  4. Connect kubectl to your AKS cluster.

    az aks get-credentials \
    --resource-group <resource-group-name> \
    --name <aks-cluster-name>

  5. Deploy Periscope.

    kubectl apply -k <path-to-kustomize-directory>

  6. Verify that the Periscope pods are running.

    kubectl get pods --namespace aks-periscope

For all available configuration values, optional Windows components, and instructions to run another collection, see the AKS Periscope Kustomize deployment guide.

Deploy Periscope using Azure Copilot

Use Azure Copilot to deploy Periscope to an AKS cluster. If Azure Copilot can't determine the cluster from the current context, it prompts you to select one. After you select the cluster, Azure Copilot might ask you to confirm the deployment details before it deploys Periscope.

Sample prompts include:

  • "Help me deploy Periscope to my AKS cluster"
  • "Deploy Periscope to my cluster"
  • "Add Periscope to my cluster"
  • "Add periscope logging to my cluster"
  • "Help me collect diagnostics logs from my AKS cluster"

Related content

  • Last updated on