Sign container images with Notation and Azure Key Vault using a self-signed certificate
Signing container images is a process that ensures their authenticity and integrity. This is achieved by adding a digital signature to the container image, which can be validated during deployment. The signature helps to verify that the image is from a trusted publisher and has not been modified. Notation is an open source supply chain security tool developed by the Notary Project community and backed by Microsoft, which supports signing and verifying container images and other artifacts. The Azure Key Vault (AKV) is used to store certificates with signing keys that can be used by Notation with the Notation AKV plugin (azure-kv) to sign and verify container images and other artifacts. The Azure Container Registry (ACR) allows you to attach signatures to container images and other artifacts as well as view those signatures.
In this tutorial:
- Install Notation CLI and AKV plugin
- Create a self-signed certificate in AKV
- Build and push a container image with ACR Tasks
- Sign a container image with Notation CLI and AKV plugin
- Validate a container image against the signature with Notation CLI
- Timestamping
Prerequisites
- Create or use an Azure Container Registry for storing container images and signatures
- Create or use an Azure Key Vault for managing certificates
- Install and configure the latest Azure CLI, or Run commands in the Azure Cloud Shell
Install Notation CLI and AKV plugin
Install Notation v1.2.0 on a Linux amd64 environment. Follow the Notation installation guide to download the package for other environments.
# Download, extract and install curl -Lo notation.tar.gz https://github.com/notaryproject/notation/releases/download/v1.2.0/notation_1.2.0_linux_amd64.tar.gz tar xvzf notation.tar.gz # Copy the Notation binary to the desired bin directory in your $PATH, for example cp ./notation /usr/local/bin
Install the Notation Azure Key Vault plugin
azure-kv
v1.2.0 on a Linux amd64 environment.Note
The URL and SHA256 checksum for the Notation Azure Key Vault plugin can be found on the plugin's release page.
notation plugin install --url https://github.com/Azure/notation-azure-kv/releases/download/v1.2.0/notation-azure-kv_1.2.0_linux_amd64.tar.gz --sha256sum 06bb5198af31ce11b08c4557ae4c2cbfb09878dfa6b637b7407ebc2d57b87b34
List the available plugins and confirm that the
azure-kv
plugin with version1.2.0
is included in the list.notation plugin ls
Configure environment variables
Note
For easy execution of commands in the tutorial, provide values for the Azure resources to match the existing ACR and AKV resources.
Configure AKV resource names.
AKV_SUB_ID=myAkvSubscriptionId AKV_RG=myAkvResourceGroup # Name of the existing AKV used to store the signing keys AKV_NAME=myakv # Name of the certificate created in AKV CERT_NAME=wabbit-networks-io CERT_SUBJECT="CN=wabbit-networks.io,O=Notation,L=Seattle,ST=WA,C=US" CERT_PATH=./${CERT_NAME}.pem
Configure ACR and image resource names.
ACR_SUB_ID=myAcrSubscriptionId ACR_RG=myAcrResourceGroup # Name of the existing registry example: myregistry.azurecr.io ACR_NAME=myregistry # Existing full domain of the ACR REGISTRY=$ACR_NAME.azurecr.io # Container name inside ACR where image will be stored REPO=net-monitor TAG=v1 IMAGE=$REGISTRY/${REPO}:$TAG # Source code directory containing Dockerfile to build IMAGE_SOURCE=https://github.com/wabbit-networks/net-monitor.git#main
Sign in with Azure CLI
az login
To learn more about Azure CLI and how to sign in with it, see Sign in with Azure CLI.
Secure access permissions to ACR and AKV
When working with ACR and AKV, it’s essential to grant the appropriate permissions to ensure secure and controlled access. You can authorize access for different entities, such as user principals, service principals, or managed identities, depending on your specific scenarios. In this tutorial, the access is authorized to a signed-in Azure user.
Authorize access to ACR
The AcrPull
and AcrPush
roles are required for signing container images in ACR.
Set the subscription that contains the ACR resource
az account set --subscription $ACR_SUB_ID
Assign the roles
USER_ID=$(az ad signed-in-user show --query id -o tsv) az role assignment create --role "AcrPull" --role "AcrPush" --assignee $USER_ID --scope "/subscriptions/$ACR_SUB_ID/resourceGroups/$ACR_RG/providers/Microsoft.ContainerRegistry/registries/$ACR_NAME"
Authorize access to AKV
In this section, we’ll explore two options for authorizing access to AKV.
Use Azure RBAC (Recommended)
The following roles are required for signing using self-signed certificates:
Key Vault Certificates Officer
for creating and reading certificatesKey Vault Certificates User
for reading existing certificatesKey Vault Crypto User
for signing operations
To learn more about Key Vault access with Azure RBAC, see Use an Azure RBAC for managing access.
Set the subscription that contains the AKV resource
az account set --subscription $AKV_SUB_ID
Assign the roles
USER_ID=$(az ad signed-in-user show --query id -o tsv) az role assignment create --role "Key Vault Certificates Officer" --role "Key Vault Crypto User" --assignee $USER_ID --scope "/subscriptions/$AKV_SUB_ID/resourceGroups/$AKV_RG/providers/Microsoft.KeyVault/vaults/$AKV_NAME"
Assign access policy in AKV (legacy)
The following permissions are required for an identity:
Create
permissions for creating a certificateGet
permissions for reading existing certificatesSign
permissions for signing operations
To learn more about assigning policy to a principal, see Assign Access Policy.
Set the subscription that contains the AKV resource:
az account set --subscription $AKV_SUB_ID
Set the access policy in AKV:
USER_ID=$(az ad signed-in-user show --query id -o tsv) az keyvault set-policy -n $AKV_NAME --certificate-permissions create get --key-permissions sign --object-id $USER_ID
Important
This example shows the minimum permissions needed for creating a certificate and signing a container image. Depending on your requirements, you may need to grant additional permissions.
Create a self-signed certificate in AKV (Azure CLI)
The following steps show how to create a self-signed certificate for testing purpose.
Create a certificate policy file.
Once the certificate policy file is executed as below, it creates a valid certificate compatible with Notary Project certificate requirement in AKV. The value for
ekus
is for code-signing, but isn't required for notation to sign artifacts. The subject is used later as trust identity that user trust during verification.cat <<EOF > ./my_policy.json { "issuerParameters": { "certificateTransparency": null, "name": "Self" }, "keyProperties": { "exportable": false, "keySize": 2048, "keyType": "RSA", "reuseKey": true }, "secretProperties": { "contentType": "application/x-pem-file" }, "x509CertificateProperties": { "ekus": [ "1.3.6.1.5.5.7.3.3" ], "keyUsage": [ "digitalSignature" ], "subject": "$CERT_SUBJECT", "validityInMonths": 12 } } EOF
Create the certificate.
az keyvault certificate create -n $CERT_NAME --vault-name $AKV_NAME -p @my_policy.json
Sign a container image with Notation CLI and AKV plugin
Authenticate to your ACR by using your individual Azure identity.
az acr login --name $ACR_NAME
Important
If you have Docker installed on your system and used az acr login
or docker login
to authenticate to your ACR, your credentials are already stored and available to notation. In this case, you don’t need to run notation login
again to authenticate to your ACR. To learn more about authentication options for notation, see Authenticate with OCI-compliant registries.
Build and push a new image with ACR Tasks. Always use the digest value to identify the image for signing since tags are mutable and can be overwritten.
DIGEST=$(az acr build -r $ACR_NAME -t $REGISTRY/${REPO}:$TAG $IMAGE_SOURCE --no-logs --query "outputImages[0].digest" -o tsv) IMAGE=$REGISTRY/${REPO}@$DIGEST
In this tutorial, if the image has already been built and is stored in the registry, the tag serves as an identifier for that image for convenience.
IMAGE=$REGISTRY/${REPO}:$TAG
Get the Key ID of the signing key. A certificate in AKV can have multiple versions, the following command gets the Key ID of the latest version.
KEY_ID=$(az keyvault certificate show -n $CERT_NAME --vault-name $AKV_NAME --query 'kid' -o tsv)
Sign the container image with the COSE signature format using the signing key ID. To sign with a self-signed certificate, you need to set the plugin configuration value
self_signed=true
.notation sign --signature-format cose --id $KEY_ID --plugin azure-kv --plugin-config self_signed=true $IMAGE
To authenticate with AKV, by default, the following credential types if enabled will be tried in order:
- Environment credential
- Workload identity credential
- Managed identity credential
- Azure CLI credential
If you want to specify a credential type, use an additional plugin configuration called
credential_type
. For example, you can explicitly setcredential_type
toazurecli
for using Azure CLI credential, as demonstrated below:notation sign --signature-format cose --id $KEY_ID --plugin azure-kv --plugin-config self_signed=true --plugin-config credential_type=azurecli $IMAGE
See below table for the values of
credential_type
for various credential types.Credential type Value for credential_type
Environment credential environment
Workload identity credential workloadid
Managed identity credential managedid
Azure CLI credential azurecli
View the graph of signed images and associated signatures.
notation ls $IMAGE
Verify a container image with Notation CLI
To verify the container image, add the root certificate that signs the leaf certificate to the trust store and create trust policies for verification. For the self-signed certificate used in this tutorial, the root certificate is the self-signed certificate itself.
Download public certificate.
az keyvault certificate download --name $CERT_NAME --vault-name $AKV_NAME --file $CERT_PATH
Add the downloaded public certificate to named trust store for signature verification.
STORE_TYPE="ca" STORE_NAME="wabbit-networks.io" notation cert add --type $STORE_TYPE --store $STORE_NAME $CERT_PATH
List the certificate to confirm.
notation cert ls
Configure trust policy before verification.
Trust policies allow users to specify fine-tuned verification policies. The following example configures a trust policy named
wabbit-networks-images
, which applies to all artifacts in$REGISTRY/$REPO
and uses the named trust store$STORE_NAME
of type$STORE_TYPE
. It also assumes that the user trusts a specific identity with the X.509 subject$CERT_SUBJECT
. For more details, see Trust store and trust policy specification.cat <<EOF > ./trustpolicy.json { "version": "1.0", "trustPolicies": [ { "name": "wabbit-networks-images", "registryScopes": [ "$REGISTRY/$REPO" ], "signatureVerification": { "level" : "strict" }, "trustStores": [ "$STORE_TYPE:$STORE_NAME" ], "trustedIdentities": [ "x509.subject: $CERT_SUBJECT" ] } ] } EOF
Use
notation policy
to import the trust policy configuration from a JSON file that we created previously.notation policy import ./trustpolicy.json notation policy show
Use
notation verify
to verify the container image hasn't been altered since build time.notation verify $IMAGE
Upon successful verification of the image using the trust policy, the sha256 digest of the verified image is returned in a successful output message.
Timestamping
Since Notation v1.2.0 release, Notation supports RFC 3161 compliant timestamping. This enhancement extends the trust of signatures created within the certificate's validity period by trusting a Timestamping Authority (TSA), enabling successful signature verification even after the certificates have expired. As an image signer, you should ensure that you sign container images with timestamps generated by a trusted TSA. As an image verifier, to verify timestamps, you should ensure that you trust both the image signer and the associated TSA, and establish trust through trust stores and trust policies. Timestamping reduces costs by eliminating the need to periodically re-sign images due to certificate expiry, which is especially critical when using short-lived certificates. For detailed instructions on how to sign and verify using timestamping, please refer to the Notary Project timestamping guide.
Next steps
Notation also provides CI/CD solutions on Azure Pipeline and GitHub Actions Workflow:
- Sign and verify a container image with Notation in Azure Pipeline
- Sign and verify a container image with Notation in GitHub Actions Workflow
To validate signed image deployment in AKS or Kubernetes: