Notation-based secure software supply chain solution for the Azure Kubernetes Service (AKS)

View GitHub repo

This solution offers a general approach to creating a Secure Software Supply Chain (SSSC) for applications hosted on the Azure Kubernetes Service (AKS). The solution focuses on improving software supply chain security for containerized workloads deployed to, and operated in, Kubernetes environments. This is achieved through defense-in-depth and trust-but-verify approaches.

The solution was developed to support requirements learned while working with many of Microsoft's largest customers. The solution architecture was developed in collaboration with Azure product engineering.

Solution architecture

This solution delivers a notation-based secure software supply chain in Azure built on Azure Kubernetes Service (AKS) and Azure Container Registry (ACR).

Notation-based implementation in AKS

Either Azure Pipelines or GitHub Actions can be used in the build phase to create the software release and security artifacts. Microsoft's sbom-tool is used to generate an SBOM from the source code, dependencies, and the container image packages. A vulnerability report is generated using Aquasec's Trivy tool (only supports Linux workloads). Notation is used to sign the container image and security artifacts with a signing cert stored in Azure Key Vault. The ORAS tool is used to bundle the signatures, security artifacts, and container image (software release), and then to publish the release bundle to Azure Container Registry.

Policy enforcement is enabled in the deploy phase by using Open Policy Agent's Gatekeeper and the Ratify verification tool. Ratify acts as an external data provider for Gatekeeper and facilitates exposing the appropriate information from the release bundle to the policy engine. Ratify verifies component signatures using a root CA certificate stored in its configuration.

The following policies are enforced:

  1. All images must be signed to ensure workload integrity.
  2. All images must have an attached and signed SBOM, and vulnerability scan result.

Notation-based secure software supply chain in Azure Kubernetes Service (AKS)

Solution overview

Overview of solution functionality

This secure software supply chain solution supports the following requirements:

  • Sign each code commit: Ensuring integrity in the software release starts when developers write the code. All code commits must be cryptographically signed in trusted development environments. Additionally, known vulnerabilities for dependencies should be surfaced in the development environment. Policy enforcement within the developer environment can be used to ensure commits are signed, new dependencies with vulnerabilities above a certain threshold are not introduced and existing dependencies are updated to the appropriate version with the fix when there are known vulnerabilities. This ensures that contributors can be cryptographically verified, and that the OSS vulnerability risk undertaken is well understood.

  • Create a component inventory: To ensure that there is a complete understanding of all the components that make up a software release, a component inventory is built as part of the build/release process. This includes all release dependencies, including third party libraries and OS packages. A list of all known component vulnerabilities is added to the inventory, providing a comprehensive understanding of the bits and vulnerabilities that make up the software release. The security team uses this information to design and implement a risk mitigation strategy.

  • Sign each release, inventory, and security artifact: To ensure that the software release, component inventory, and vulnerability collection can't be manipulated by untrusted actors, the software supply chain security artifacts are cryptographically signed in a trusted environment. This ensures that the integrity of all of these artifacts can be cryptographically verified later, even in low-trust environments.

  • Assure component integrity with attestations: In addition, attestations are used to provide cryptographically verifiable traceability. For example, an attestation can confirm that a component was built by an authorized build runner, backed by a VM running a known Azure subscription, and signed at a specific date-time. Signatures provide confidence in the integrity of all released artifacts, and attestations provide traceability.

  • Enforce security policies: The software supply chain security artifacts contain useful information, but need to be combined with policy enforcement to ensure that all components are properly signed and can run with acceptable risk within the software development and deployment lifecycle. Examples include not allowing a software release to be built when a dependency has a major vulnerability and only allowing signed container images to be deployed to Kubernetes clusters.

  • Enable developers to observe supply chain knowledge graph: It is important to provide developers with the tools needed to understand the relationships between the software supply chain artifacts as well as the systems and processes that touch them. An emerging best practice is to enable developers to observe these components and relationships via a knowledge graph. If you learn about a new vulnerability identified in a particular release of log4j, you might ask questions like: "Which of my recent releases have log4j version 2.12.3 as a dependency?" Another question might be, "Which software releases deployed to which clusters have exposure to CVE-2021-44228?"

  • For more information

  • Notary Project: Standards-based spec and tooling for securing software supply chains

  • Sigstore: Sign. verify. protect. Making sure your software is what it claims to be.

  • Linux Foundation: What is a Software Bill of Materials (SBOM) ?

  • Cloud Native Computing Foundation (CNCF): A MAP for Kubernetes supply chain security

  • MS Learn: Introduction to Microsoft's framework for securing containers