Edit

Use ImageCleaner to clean up stale images on your Azure Kubernetes Service cluster (preview)

  • Article
  • 4 minutes to read

It's common to use pipelines to build and deploy images on Azure Kubernetes Service (AKS) clusters. While great for image creation, this process often doesn't account for the stale images left behind and can lead to image bloat on cluster nodes. These images can present security issues as they may contain vulnerabilities. By cleaning these unreferenced images, you can remove an area of risk in your clusters. When done manually, this process can be time intensive, which ImageCleaner can mitigate via automatic image identification and removal.

Note

ImageCleaner is a feature based on Eraser. On an AKS cluster, the feature name and property name is ImageCleaner while the relevant ImageCleaner pods' names contain Eraser.

Important

AKS preview features are available on a self-service, opt-in basis. Previews are provided "as is" and "as available," and they're excluded from the service-level agreements and limited warranty. AKS previews are partially covered by customer support on a best-effort basis. As such, these features aren't meant for production use. For more information, see the following support articles:

Prerequisites

  • An Azure subscription. If you don't have an Azure subscription, you can create a free account.
  • Azure CLI or Azure PowerShell and the aks-preview 0.5.96 or later CLI extension installed.
  • The EnableImageCleanerPreview feature flag registered on your subscription:

First, install the aks-preview extension by running the following command:

az extension add --name aks-preview

Run the following command to update to the latest version of the extension released:

az extension update --name aks-preview

Then register the EnableImageCleanerPreview feature flag by using the az feature register command, as shown in the following example:

az feature register --namespace "Microsoft.ContainerService" --name "EnableImageCleanerPreview"

It takes a few minutes for the status to show Registered. Verify the registration status by using the az feature show command:

az feature show --namespace "Microsoft.ContainerService" --name "EnableImageCleanerPreview"

When the status reflects Registered, refresh the registration of the Microsoft.ContainerService resource provider by using the az provider register command:

az provider register --namespace Microsoft.ContainerService

Limitations

ImageCleaner does not support the following:

How ImageCleaner works

When enabled, an eraser-controller-manager pod is deployed on each agent node, which will use an ImageList CRD to determine unreferenced and vulnerable images. Vulnerability is determined based on a trivy scan, after which images with a LOW, MEDIUM, HIGH, or CRITICAL classification are flagged. An updated ImageList will be automatically generated by ImageCleaner based on a set time interval, and can also be supplied manually.

Once an ImageList is generated, ImageCleaner will remove all the images in the list from node VMs.

A diagram showing ImageCleaner's workflow. The ImageCleaner pods running on the cluster can generate an ImageList, or manual input can be provided.

Configuration options

In addition to choosing between manual and automatic mode, there are several options for ImageCleaner:

Name Description Required
--enable-image-cleaner Enable the ImageCleaner feature for an AKS cluster Yes, unless disable is specified
--disable-image-cleaner Disable the ImageCleaner feature for an AKS cluster Yes, unless enable is specified
--image-cleaner-interval-hours This parameter determines the interval time (in hours) ImageCleaner will use to run. The default value for Azure CLI is one week, the minimum value is 24 hours and the maximum is three months. Not required for Azure CLI, required for ARM template or other clients

Note

After disabling ImageCleaner, the old configuration still exists. This means that if you enable the feature again without explicitly passing configuration, the existing value will be used rather than the default.

Enable ImageCleaner on your AKS cluster

To create a new AKS cluster using the default interval, use az aks create:

az aks create -g MyResourceGroup -n MyManagedCluster \ 
  --enable-image-cleaner

To enable on an existing AKS cluster, use az aks update:

az aks update -g MyResourceGroup -n MyManagedCluster \ 
  --enable-image-cleaner

The --image-cleaner-interval-hours parameter can be specified at creation time or for an existing cluster. For example, the following command updates the interval for a cluster with ImageCleaner already enabled:

az aks update -g MyResourceGroup -n MyManagedCluster \
  --image-cleaner-interval-hours 48

After the feature is enabled, the eraser-controller-manager-xxx pod and collector-aks-xxx pod will be deployed. Based on your configuration, ImageCleaner will generate an ImageList containing non-running and vulnerable images at the desired interval. ImageCleaner will automatically remove these images from cluster nodes.

Manually remove images

To manually remove images from your cluster using ImageCleaner, first create an ImageList. For example, save the following as image-list.yml:

apiVersion: eraser.sh/v1alpha1
kind: ImageList
metadata:
  name: imagelist
spec:
  images:
    - docker.io/library/alpine:3.7.3   # You can also use "*" to specify all non-running images

And apply it to the cluster:

kubectl apply -f image-list.yml

A job named eraser-aks-xxxwill be triggered which causes ImageCleaner to remove the desired images from all nodes.

Disable ImageCleaner

To stop using ImageCleaner, you can disable it via the --disable-image-cleaner flag:

az aks update -g MyResourceGroup -n MyManagedCluster
  --disable-image-cleaner

Logging

The deletion logs are stored in the image-cleaner-kind-worker pods. You can check these via kubectl logs or via the Container Insights pod log table if the Azure Monitor add-on is enabled.