Overview of the Run stage

  • Article

The Run stage is the fifth stage of the Containers Secure Supply Chain (CSSC) framework. This stage emphasizes scanning and monitoring the runtime environments and purging them of outdated and vulnerable images. This overview provides the background, objectives, and goals for the Run stage of the CSSC framework.

Microsoft's Containers Secure Supply Chain (CSSC) framework identifies the need for running containers with trusted images and provides a set of best practices and tools to help securely run images and reduce the attack surface of runtime. In this article, you'll learn about the objectives, best practices, and tools that you can use in the Run stage of the CSSC framework.

Background

Currently, enterprises use various approaches to run compliant containerized workloads with trusted images. Monitoring deployed workloads provides enterprises with validation that the true operational state is the expected state. Workload images will become vulnerable at the time of, or after they are deployed. Enterprises are suggested to continuously scan their runtime environments and images to detect which workloads are now vulnerable and which images are out of support to receive security updates or bug fixes.

The Run stage of the CSSC framework recommends a set of steps and security controls that should be implemented to ensure running containers and runtime hosts are secure, such as recycling nodes on timely manner, upgrading the containers with up-to-date and patched container images, removing the stale, non-running container images, and preventing undesirable behavior of the containers.

Microsoft recommends continuously running vulnerability and malware scanner for containerized workloads and runtime. Regularly updating the containers and the nodes as well as keeping the nodes clean are effective practices to protect containerized applications from compromising.

  • Regularly scan for vulnerability and malware, and check the image lifecycle metadata to identify images that need to be patched and updated. Regularly clean up stale images from the cache on the node to reduce the likelihood a vulnerable stale images can be used by bad actors
  • Configure strong authentication and authorization mechanisms on hosting environments and containers, as well as running containers as non-root as attackers can't access the systems with ease and cause damage upon compromise
  • Regularly update containers and work nodes. This will ensure that the containers and the nodes are running with the latest security patches and fixes
  • Reduce attack surface by restricting container and node port, restricting network access of containers, enable mutual TLS.
  • Enforce resource constraints to containers, such as control how much memory, or CPU a container can use, to mitigate the risk of system instability
  • Follow industry standard guidance like CIS benchmarks, CISE guidance, CNCF Software Supply Chain Best Practices, NIST guidance, or regional government guidance based on your needs

Workflow for continuously scanning and monitoring runtime environments

The Run stage has a workflow in place to continuously scan and monitor runtime environments. The Run stage workflow applies to purge the vulnerable and outdated container images. It is very crucial to keep the runtime environments secure, the workflow follows these steps:

  1. Continuously scan for vulnerabilities and malware in containerized workloads and runtime environments to check for any potential security threats.
  2. Regularly update containers and worker nodes to ensure that they are running with the latest security patches and fixes.
  3. Regularly update the containers and the nodes to protect containerized applications from compromise and avoid the risk of vulnerabilities from patches and fixes.
  4. Check the image lifecycle metadata to identify the images that needs an upgrade to be latest and secure.
  5. Regularly clean up stale images from the cache on the node to avoid vulnerable stale images being used by bad actors.
  6. Configure strong authentication and authorization mechanisms on hosting environments and containers, as well as running containers to prevent attackers from accessing the systems with ease and causing damage upon compromise.
  7. Reduce attack surface by restricting container and node port, restricting network access of containers, enabling mutual TLS, and enforcing resource constraints to containers, such as controlling how much memory or CPU a container can use, to mitigate the risk of system instability.

Security goals in the Run stage

The Run stage of the CSSC framework is intended to satisfy the following security goals.

Monitor runtime to reduce running vulnerable images

Scan containers for vulnerabilities and compliance with organization policies. Verify if the containers are using the latest version of images.

Keeping your runtime containers up to date ensures containers are always free of vulnerabilities and compliant to organization policies. Images should be continuously monitored throughout the stages. New images from the Acquire stage or Build stage can trigger the update of runtime containers in the Run stage. The images can be updated for various reasons, like fixing vulnerabilities, fixing the software that license becomes non-compliant, and the image becoming end-of-support over time. All of these updates will trigger the runtime containers to be updated.

Prevent non-compliant images and clean up stale images to minimize the attack risk

It's common for CI/CD pipelines to build and push images to the deployment platform in the Deploy stage often, but unused images on a runtime node may not be purged reguarly. This can lead to accumulating bloat on the disk, and a host of non-compliant images lingering on the nodes. Vulnerabilities are also likely to exist in stale images. Regular clean up of stale images can avoid unnecessary scanning and reduce the attack surface of the runtime environment.

Keep hosting environment up-to-date and with secure configurations

Keep hosting environment up-to-date with the security releases and patches from the trusted upstream or cloud provider. Ensure strict access control and limited network permission to reduce attack surface of the runtime environments. Adopt real-time detection of unexpected behavior, misconfiguration, and attacks on hosting environment.

Microsoft offers a set of tools and services that can help enterprises implement the recommended steps in the Run stage workflow and address the security goals listed above.

Tools and services for vulnerability scanning and patching images

Microsoft Defender for Cloud is the cloud-native solution to improve, monitor, and maintain the security of your containerized workloads. Microsoft Defender for Cloud offers vulnerability assessment and management tools for images stored in Azure Container Registry and running containers.

Tools and services for cleaning non-compliant images

Azure Image Cleaner performs automatic image identification and removal. Use Azure Image Cleaner to clean up stale images from Kubernetes nodes for AKS container workloads, or use the open-source Eraser for non-AKS or vanilla Kubernetes environment, which mitigates the risk of stale images and reduces the time required to clean them up.

Tools for automatically upgrade runtime service

Cluster auto-upgrade provides a "set once and forget" mechanism that yields tangible time and operational cost benefits. Enable AKS Auto-Upgrade ensures your clusters are up to date and don't miss the security releases or patches from AKS and upstream Kubernetes if you are using AKS.

Next steps

See overview of the Observability stage for securely observe the containers and locate potential supply chain security issues in time.