Overview of the Build stage

  • Article

Building container images is the third stage of the Containers Secure Supply Chain (CSSC) framework. While some enterprises pull the base images from external sources directly, others may pull the base images from the internal catalog without verifying the images against their signatures and attestations. It is crucial to ensure that base images are always pulled from the internal catalog and verified before use. After building the images, some enterprises publish them without vulnerability and malware scanning, and fail to generate attestations like SBOM, which cannot be verified in subsequent supply chain stages. It is essential to ensure that the produced container images are trusted and compliant with the enterprise policies.

Microsoft's Containers Secure Supply Chain (CSSC) framework identifies the need for building container images and provides a set of best practices and tools to help you securely build trusted and compliant images. In this article, you'll learn about the objectives, best practices, and tools that you can use in the Build stage of the CSSC framework.

Background

Currently, enterprises use various approaches to build compliant container images securely. Before building container images, internal teams are required to pull base images from internal golden registries. However, these images are not verified before use, which is not recommended as golden registries can be compromised and images can become malicious. Enterprises also control the source of libraries and packages that are added into the container images to ensure quality.

After building container images, enterprises generate additional metadata to ensure that the container images are compliant. Some enterprises leverage a document of SLSA provenance to know which git commit is responsible for producing the container image and whether it is reproducible. Others use Docker labels to provide provenance on the owner of the container image and who is responsible for the commit of the image. Some enterprises scan the image for vulnerabilities and malware and generate reports accordingly.

The Build stage of the CSSC framework recommends a set of steps and security controls that should be implemented to produce trustworthy and compliant container images before they are published.

Microsoft recommends the following practices for building compliant container images securely.

  • Pull base images only from the internal catalog to reduce the attack surface and improve security.
  • When available, verify signatures and attestations of base images before using them to ensure they are trustworthy and compliant.
  • Distroless images contain only the application and its runtime dependencies, without any additional software or operating system components. Use distroless images as base images to reduce the attack surface and improve security.
  • Apply least privilege access to the build platform or service to ensure that only authorized users are able to make changes or modifications, thus reducing the risk of unauthorized access, data breaches, and other security incidents.
  • Scan container images for vulnerabilities and malware post-build to mitigate the risk of introducing security vulnerabilities.
  • Enrich container images after the build with additional metadata like SBOM and provenance metadata.
  • Sign container images and relevant metadata after the build with enterprise keys to ensure integrity and provide trusted stamp of approval for internal use.
  • Patch container images in a timely manner to address known vulnerabilities.
  • Keep dependencies reasonably up-to-date to keep up with the vulnerabilities and security fixes.

Workflow for building compliant container images

The Build stage has a workflow in place to create a container image compliant with organizations and security policies. It is crucial to apply privilege level access control to the build platform or service for authorized users only. When base images are available in the Catalog, the Build workflow does the following:

  1. Pull base images from the internal Catalog only.
  2. Verify base images before using it to ensure it is trustworthy and compliant.
  3. Add additional frameworks, application code, and/or configurations from trust source on top of base images for build.
  4. Generate SBOM during the build process.
  5. Scan the resulting container image for known vulnerabilities and malware after build to prevent introduction of security flaws.
  6. Patch the resulting container image to address known vulnerabilities and malware
  7. Attach the vulnerability and malware reports as image attestations to use in subsequent stages of the supply chain.
  8. Attach SBOM as image attestations to use in subsequent stages of the supply chain.
  9. Enrich the resulting image with additional metadata like provenance metadata to use in subsequent stages of the supply chain.
  10. Sign the resulting image and relevant metadata with enterprise keys to ensure integrity and provide trusted stamp of approval for the Deployment stage

Security goals in the Build stage

Having a well-defined workflow for building trustworthy and compliant container images helps enterprises increase their security and reduce the attack surface on their supply chain for containers. The Build stage of the CSSC framework is intended to satisfy the following security goals.

Reduce attack surface due to improper access control

Internal users may make unexpected changes or incorrect configurations on build infrastructure. The Build stage in the CSSC framework addresses this risk by enforcing least privilege access control by default, so that resource owners can ensure that the right person has the right permissions to make changes and configurations, thus reducing attack surface.

Reduce attack surface from malicious internal users

Identity theft can be used to access the build platform, modify binaries, and inject illicit changes that are invisible to the original developer and users. There is also a risk of injecting vulnerable code or misconfiguration by internal users. The Build stage in the CSSC framework addresses this risk by applying Azure role-based access control (Azure RBAC) and enforce code review from various code owners.

Reduce attack surface due to external dependencies

Base images from external sources can be malicious. External registries can be compromised. The Build stage in the CSSC framework addresses this risk by using base images from the catalog only and verifying them against their signatures, vulnerability and malware reports, and other image metadata before using them.

Malicious Open Source Software (OSS) packages with hidden backdoors can cause the package’s binaries to not match with the source code. The validity of dependencies or open-source components can expire, blocking the latest security fixes from upstream. The Build stage in the CSSC framework addresses this risk by using approved libraries, scanning for vulnerabilities and malware on a regular basis, and keeping libraries up to date.

Reduce attack surface due to compromised builds

Vulnerable build agents can inject side binaries or malicious code that can cause attacks like the SolarWinds attack. The Build stage in the CSSC framework addresses this risk by patching the build agents on a regular basis.

Ensure a minimum image quality

In the Build stage of the CSSC framework, enterprises can configure required steps for vulnerability and malware scanning of the images after build, and generate reports. Enterprises can also enrich the images after build with additional metadata including SBOMs, provenance metadata, and other internal metadata that meets their policies. They can also sign the artifacts with enterprise keys to ensure integrity and provide a trusted stamp of approval for the Deployment stage.

Microsoft offers a set of tools and services that can help enterprises implement the recommended steps in the Build stage workflow and address the security goals listed above.

Tools for staying up-to-date after build

Dependabot is a tool for automating dependency updates through pull requests. For details and next steps, please refer to this guide.

Copa is a Microsoft-backed open-source project, addresses Linux OS package vulnerabilities in container images based on vulnerability scanning results. For details and next steps, please refer to this guide.

Tools and service for continuous vulnerability scanning and patching images

Microsoft Defender for Cloud is the cloud-native solution to improve, monitor, and maintain the security of your containerized workloads. Microsoft Defender for Cloud offers vulnerability assessment and management tools for images stored in Azure Container Registry.

Copa is a Microsoft-backed open-source project, addresses Linux OS package vulnerabilities in container images based on vulnerability scanning results. For details and next steps, please refer to this guide.

Tools and services for enriching external images with enterprise metadata

ORAS is a CNCF project backed by Microsoft that offers an open-source CLI and libraries to interact with OCI registries. ORAS can be used to add annotations and attach metadata to a container image. ORAS can be used to add SBOMs, provenance metadata and other metadata to enrich the images for subsequent stages in the supply chain.

Microsoft's SBOM Tool is an open-source, highly scalable, and enterprise-ready tool to create SPDX 2.2 compatible SBOMs for any variety of artifacts. SBOM Tool can be used to generate detailed SBOMs for container images.

Notary Project is a Microsoft-backed CNCF project that develops specifications and tooling for signing and verifying software artifacts. Notary Project's notation tool can be used to sign container images and other cloud-native artifacts with enterprise keys.

Azure Key Vault is a cloud-based service that provides a secure and centralized location for storing and managing cryptographic keys, secrets, and certificates. AKV can be used for storing and managing your enterprise keys securely and reliably.

Next steps

See overview of the Deploy stage to securely deploy container images.