Use Image Cleaner to clean up stale images on your Azure Kubernetes Service (AKS) cluster
It's common to use pipelines to build and deploy images on Azure Kubernetes Service (AKS) clusters. While great for image creation, this process often doesn't account for the stale images left behind and can lead to image bloat on cluster nodes. These images may contain vulnerabilities, which may create security issues. To remove security risks in your clusters, you can clean these unreferenced images. Manually cleaning images can be time intensive. Image Cleaner performs automatic image identification and removal, which mitigates the risk of stale images and reduces the time required to clean them up.
Note
Image Cleaner is a feature based on Eraser.
On an AKS cluster, the feature name and property name is Image Cleaner
, while the relevant Image Cleaner pods' names contain Eraser
.
Prerequisites
- An Azure subscription. If you don't have an Azure subscription, you can create a free account.
- Azure CLI version 2.49.0 or later. Run
az --version
to find your version. If you need to install or upgrade, see Install Azure CLI.
Limitations
Image Cleaner doesn't yet support Windows node pools or AKS virtual nodes.
How Image Cleaner works
When you enable Image Cleaner, it deploys an eraser-controller-manager
pod, which generates an ImageList
CRD. The eraser pods running on each node clean up any unreferenced and vulnerable images according to the ImageList
. A trivy scan helps determine vulnerability and flags images with a classification of LOW
, MEDIUM
, HIGH
, or CRITICAL
. Image Cleaner automatically generates an updated ImageList
based on a set time interval and can also be supplied manually. Once Image Cleaner generates an ImageList
, it removes all images in the list from node VMs.
Configuration options
With Image Cleaner, you can choose between manual and automatic mode and the following configuration options:
Name | Description | Required |
---|---|---|
--enable-image-cleaner |
Enable the Image Cleaner feature for an AKS cluster | Yes, unless disable is specified |
--disable-image-cleaner |
Disable the Image Cleaner feature for an AKS cluster | Yes, unless enable is specified |
--image-cleaner-interval-hours |
This parameter determines the interval time (in hours) Image Cleaner uses to run. The default value for Azure CLI is one week, the minimum value is 24 hours and the maximum is three months. | Not required for Azure CLI, required for ARM template or other clients |
Note
After disabling Image Cleaner, the old configuration still exists. This means if you enable the feature again without explicitly passing configuration, the existing value is used instead of the default.
Enable Image Cleaner on your AKS cluster
Enable Image Cleaner on a new cluster
Enable Image Cleaner on a new AKS cluster using the
az aks create
command with the--enable-image-cleaner
parameter.az aks create -g myResourceGroup -n myManagedCluster \ --enable-image-cleaner
Enable Image Cleaner on an existing cluster
Enable Image Cleaner on an existing AKS cluster using the
az aks update
command.az aks update -g myResourceGroup -n myManagedCluster \ --enable-image-cleaner
Update the Image Cleaner interval on a new or existing cluster
Update the Image Cleaner interval on a new or existing AKS cluster using the
--image-cleaner-interval-hours
parameter.# Update the interval on a new cluster az aks create -g myResourceGroup -n myManagedCluster \ --enable-image-cleaner \ --image-cleaner-interval-hours 48 # Update the interval on an existing cluster az aks update -g myResourceGroup -n myManagedCluster \ --image-cleaner-interval-hours 48
After you enable the feature, the eraser-controller-manager-xxx
pod and collector-aks-xxx
pod are deployed. The eraser-aks-xxx
pod contains three containers:
- Scanner container: Performs vulnerability image scans
- Collector container: Collects nonrunning and unused images
- Remover container: Removes these images from cluster nodes
Image Cleaner generates an ImageList
containing nonrunning and vulnerable images at the desired interval based on your configuration. Image Cleaner automatically removes these images from cluster nodes.
Manually remove images using Image Cleaner
Create an
ImageList
using the following example YAML namedimage-list.yml
.apiVersion: eraser.sh/v1alpha1 kind: ImageList metadata: name: imagelist spec: images: - docker.io/library/alpine:3.7.3 // You can also use "*" to specify all non-running images
Apply the
ImageList
to your cluster using thekubectl apply
command.kubectl apply -f image-list.yml
Applying the
ImageList
triggers a job namederaser-aks-xxx
, which causes Image Cleaner to remove the desired images from all nodes. Unlike theeraser-aks-xxx
pod under autoclean with three containers, the eraser-pod here has only one container.
Image exclusion list
Images specified in the exclusion list aren't removed from the cluster. Image Cleaner supports system and user-defined exclusion lists. It's not supported to edit the system exclusion list.
Check the system exclusion list
Check the system exclusion list using the following
kubectl get
command.kubectl get -n kube-system cm eraser-system-exclusion -o yaml
Create a user-defined exclusion list
Create a sample JSON file to contain excluded images.
cat > sample.json <<EOF {"excluded": ["excluded-image-name"]} EOF
Create a
configmap
using the sample JSON file using the followingkubectl create
andkubectl label
command.kubectl create configmap excluded --from-file=sample.json --namespace=kube-system kubectl label configmap excluded eraser.sh/exclude.list=true -n kube-system
Verify the images are in the exclusion list using the following
kubectl logs
command.kubectl logs -n kube-system <eraser-pod-name>
Image Cleaner image logs
Deletion image logs are stored in eraser-aks-nodepool-xxx
pods for manually deleted images and in collector-aks-nodes-xxx
pods for automatically deleted images.
You can view these logs using the kubectl logs <pod name> -n kubesystem
command. However, this command may return only the most recent logs, since older logs are routinely deleted. To view all logs, follow these steps to enable the Azure Monitor add-on and use the Container Insights pod log table.
Ensure Azure Monitoring is enabled on your cluster. For detailed steps, see Enable Container Insights on AKS clusters.
Get the Log Analytics resource ID using the
az aks show
command.az aks show -g myResourceGroup -n myManagedCluster
After a few minutes, the command returns JSON-formatted information about the solution, including the workspace resource ID.
"addonProfiles": { "omsagent": { "config": { "logAnalyticsWorkspaceResourceID": "/subscriptions/<WorkspaceSubscription>/resourceGroups/<DefaultWorkspaceRG>/providers/Microsoft.OperationalInsights/workspaces/<defaultWorkspaceName>" }, "enabled": true } }
In the Azure portal, search for the workspace resource ID, then select Logs.
Copy this query into the table, replacing
name
with eithereraser-aks-nodepool-xxx
(for manual mode) orcollector-aks-nodes-xxx
(for automatic mode).let startTimestamp = ago(1h); KubePodInventory | where TimeGenerated > startTimestamp | project ContainerID, PodName=Name, Namespace | where PodName contains "name" and Namespace startswith "kube-system" | distinct ContainerID, PodName | join ( ContainerLog | where TimeGenerated > startTimestamp ) on ContainerID // at this point before the next pipe, columns from both tables are available to be "projected". Due to both // tables having a "Name" column, we assign an alias as PodName to one column which we actually want | project TimeGenerated, PodName, LogEntry, LogEntrySource | summarize by TimeGenerated, LogEntry | order by TimeGenerated desc
Select Run. Any deleted image logs appear in the Results area.
Disable Image Cleaner
Disable Image Cleaner on your cluster using the
az aks update
command with the--disable-image-cleaner
parameter.az aks update -g myResourceGroup -n myManagedCluster \ --disable-image-cleaner
Feedback
Submit and view feedback for