Events
Mar 17, 11 PM - Mar 21, 11 PM
Join the meetup series to build scalable AI solutions based on real-world use cases with fellow developers and experts.
Register nowThis browser is no longer supported.
Upgrade to Microsoft Edge to take advantage of the latest features, security updates, and technical support.
Signing and verifying container images with a certificate issued by a trusted Certificate Authority (CA) is a valuable security practice. This security measure will help you to responsibly identify, authorize, and validate the identity of both the publisher of the container image and the container image itself. The Trusted Certificate Authorities (CAs) such as GlobalSign, DigiCert, and others play a crucial role in the validation of a user's or organization's identity, maintaining the security of digital certificates, and revoking the certificate immediately upon any risk or misuse.
Here are some essential components that help you to sign and verify container images with a certificate issued by a trusted CA:
When you verify the image, the signature is used to validate the integrity of the image and the identity of the signer. This helps to ensure that the container images are not tampered with and are from a trusted source.
In this article:
Note
We recommend creating a new Azure Key Vault for storing certificates only.
Install Notation v1.3.0 on a Linux amd64 environment. Follow the Notation installation guide to download the package for other environments.
# Download, extract and install
curl -Lo notation.tar.gz https://github.com/notaryproject/notation/releases/download/v1.3.0/notation_1.3.0_linux_amd64.tar.gz
tar xvzf notation.tar.gz
# Copy the notation cli to the desired bin directory in your PATH, for example
cp ./notation /usr/local/bin
Install the Notation Azure Key Vault plugin azure-kv
v1.2.1 on a Linux amd64 environment.
Note
The URL and SHA256 checksum for the Notation Azure Key Vault plugin can be found on the plugin's release page.
notation plugin install --url https://github.com/Azure/notation-azure-kv/releases/download/v1.2.1/notation-azure-kv_1.2.1_linux_amd64.tar.gz --sha256sum 67c5ccaaf28dd44d2b6572684d84e344a02c2258af1d65ead3910b3156d3eaf5
List the available plugins and confirm that the azure-kv
plugin with version 1.2.1
is included in the list.
notation plugin ls
Note
This guide uses environment variables for convenience when configuring the AKV and ACR. Update the values of these environment variables for your specific resources.
Configure environment variables for AKV and certificates
AKV_SUB_ID=myAkvSubscriptionId
AKV_RG=myAkvResourceGroup
AKV_NAME=myakv
# Name of the certificate created or imported in AKV
CERT_NAME=wabbit-networks-io
# X.509 certificate subject
CERT_SUBJECT="CN=wabbit-networks.io,O=Notation,L=Seattle,ST=WA,C=US"
Configure environment variables for ACR and images.
ACR_SUB_ID=myAcrSubscriptionId
ACR_RG=myAcrResourceGroup
# Name of the existing registry example: myregistry.azurecr.io
ACR_NAME=myregistry
# Existing full domain of the ACR
REGISTRY=$ACR_NAME.azurecr.io
# Container name inside ACR where image will be stored
REPO=net-monitor
TAG=v1
# Source code directory containing Dockerfile to build
IMAGE_SOURCE=https://github.com/wabbit-networks/net-monitor.git#main
az login
To learn more about Azure CLI and how to sign in with it, see Sign in with Azure CLI.
When creating certificates for signing and verification, the certificates must meet the Notary Project certificate requirement.
Here are the requirements for root and intermediate certificates:
basicConstraints
extension must be present and marked as critical. The CA
field must be set true
.keyUsage
extension must be present and marked critical
. Bit positions for keyCertSign
MUST be set.Here are the requirements for certificates issued by a CA:
CN
), country (C
), state or province (ST
), and organization (O
). In this tutorial, $CERT_SUBJECT
is used as the subject.DigitalSignature
only.1.3.6.1.5.5.7.3.3
(for Codesigning).exportable
property must be set to false
.Important
To ensure successful integration with Image Integrity, the content type of certificate should be set to PEM.
Note
This guide uses version 1.0.1 of the AKV plugin. Prior versions of the plugin had a limitation that required a specific certificate order in a certificate chain. Version 1.0.1 of the plugin does not have this limitation so it is recommended that you use version 1.0.1 or later.
Create a certificate signing request (CSR) by following the instructions in create certificate signing request.
Important
When merging the CSR, make sure you merge the entire chain that brought back from the CA vendor.
To import the certificate:
Note
If the certificate does not contain a certificate chain after creation or importing, you can obtain the intermediate and root certificates from your CA vendor. You can ask your vendor to provide you with a PEM file that contains the intermediate certificates (if any) and root certificate. This file can then be used at step 5 of signing container images.
When working with ACR and AKV, it’s essential to grant the appropriate permissions to ensure secure and controlled access. You can authorize access for different entities, such as user principals, service principals, or managed identities, depending on your specific scenarios. In this tutorial, the access are authorized to a signed-in Azure user.
The AcrPull
and AcrPush
roles are required for building and signing container images in ACR.
Set the subscription that contains the ACR resource
az account set --subscription $ACR_SUB_ID
Assign the roles
USER_ID=$(az ad signed-in-user show --query id -o tsv)
az role assignment create --role "AcrPull" --role "AcrPush" --assignee $USER_ID --scope "/subscriptions/$ACR_SUB_ID/resourceGroups/$ACR_RG/providers/Microsoft.ContainerRegistry/registries/$ACR_NAME"
Authenticate to your ACR by using your individual Azure identity.
az acr login --name $ACR_NAME
Important
If you have Docker installed on your system and used az acr login
or docker login
to authenticate to your ACR, your credentials are already stored and available to notation. In this case, you don’t need to run notation login
again to authenticate to your ACR. To learn more about authentication options for notation, see Authenticate with OCI-compliant registries.
Build and push a new image with ACR Tasks. Always use digest
to identify the image for signing, since tags are mutable and can be overwritten.
DIGEST=$(az acr build -r $ACR_NAME -t $REGISTRY/${REPO}:$TAG $IMAGE_SOURCE --no-logs --query "outputImages[0].digest" -o tsv)
IMAGE=$REGISTRY/${REPO}@$DIGEST
In this tutorial, if the image has already been built and is stored in the registry, the tag serves as an identifier for that image for convenience.
IMAGE=$REGISTRY/${REPO}@$TAG
Set the subscription that contains the AKV resource
az account set --subscription $AKV_SUB_ID
Assign the roles
If the certificate contains the entire certificate chain, the principal must be assigned with the following roles:
Key Vault Secrets User
for reading secretsKey Vault Certificates User
for reading certificatesKey Vault Crypto User
for signing operationsUSER_ID=$(az ad signed-in-user show --query id -o tsv)
az role assignment create --role "Key Vault Secrets User" --role "Key Vault Certificates User" --role "Key Vault Crypto User" --assignee $USER_ID --scope "/subscriptions/$AKV_SUB_ID/resourceGroups/$AKV_RG/providers/Microsoft.KeyVault/vaults/$AKV_NAME"
If the certificate doesn't contain the chain, the principal must be assigned with the following roles:
Key Vault Certificates User
for reading certificatesKey Vault Crypto User
for signing operationsUSER_ID=$(az ad signed-in-user show --query id -o tsv)
az role assignment create --role "Key Vault Certificates User" --role "Key Vault Crypto User" --assignee $USER_ID --scope "/subscriptions/$AKV_SUB_ID/resourceGroups/$AKV_RG/providers/Microsoft.KeyVault/vaults/$AKV_NAME"
To learn more about Key Vault access with Azure RBAC, see Use an Azure RBAC for managing access.
To set the subscription that contains the AKV resources, run the following command:
az account set --subscription $AKV_SUB_ID
If the certificate contains the entire certificate chain, the principal must be granted key permission Sign
, secret permission Get
, and certificate permissions Get
. To grant these permissions to the principal:
USER_ID=$(az ad signed-in-user show --query id -o tsv)
az keyvault set-policy -n $AKV_NAME --key-permissions sign --secret-permissions get --certificate-permissions get --object-id $USER_ID
If the certificate doesn't contain the chain, the principal must be granted key permission Sign
, and certificate permissions Get
. To grant these permissions to the principal:
USER_ID=$(az ad signed-in-user show --query id -o tsv)
az keyvault set-policy -n $AKV_NAME --key-permissions sign --certificate-permissions get --object-id $USER_ID
To learn more about assigning policy to a principal, see Assign Access Policy.
Get the Key ID for a certificate. A certificate in AKV can have multiple versions, the following command gets the Key ID for the latest version of the $CERT_NAME
certificate.
KEY_ID=$(az keyvault certificate show -n $CERT_NAME --vault-name $AKV_NAME --query 'kid' -o tsv)
Sign the container image with the COSE signature format using the Key ID.
If the certificate contains the entire certificate chain, run the following command:
notation sign --signature-format cose $IMAGE --id $KEY_ID --plugin azure-kv
If the certificate does not contain the chain, use the --plugin-config ca_certs=<ca_bundle_file>
parameter to pass the CA certificates in a PEM file to AKV plugin, run the following command:
notation sign --signature-format cose $IMAGE --id $KEY_ID --plugin azure-kv --plugin-config ca_certs=<ca_bundle_file>
To authenticate with AKV, by default, the following credential types if enabled will be tried in order:
If you want to specify a credential type, use an additional plugin configuration called credential_type
. For example, you can explicitly set credential_type
to azurecli
for using Azure CLI credential, as demonstrated below:
notation sign --signature-format cose --id $KEY_ID --plugin azure-kv --plugin-config credential_type=azurecli $IMAGE
See below table for the values of credential_type
for various credential types.
Credential type | Value for credential_type |
---|---|
Environment credential | environment |
Workload identity credential | workloadid |
Managed identity credential | managedid |
Azure CLI credential | azurecli |
View the graph of signed images and associated signatures.
notation ls $IMAGE
In the following example of output, a signature of type application/vnd.cncf.notary.signature
identified by digest sha256:d7258166ca820f5ab7190247663464f2dcb149df4d1b6c4943dcaac59157de8e
is associated to the $IMAGE
.
myregistry.azurecr.io/net-monitor@sha256:17cc5dd7dfb8739e19e33e43680e43071f07497ed716814f3ac80bd4aac1b58f
└── application/vnd.cncf.notary.signature
└── sha256:d7258166ca820f5ab7190247663464f2dcb149df4d1b6c4943dcaac59157de8e
Add the root certificate to a named trust store for signature verification. If you do not have the root certificate, you can obtain it from your CA. The following example adds the root certificate $ROOT_CERT
to the $STORE_NAME
trust store.
STORE_TYPE="ca"
STORE_NAME="wabbit-networks.io"
notation cert add --type $STORE_TYPE --store $STORE_NAME $ROOT_CERT
List the root certificate to confirm the $ROOT_CERT
is added successfully.
notation cert ls
Configure trust policy before verification.
Trust policies allow users to specify fine-tuned verification policies. Use the following command to configure trust policy.
cat <<EOF > ./trustpolicy.json
{
"version": "1.0",
"trustPolicies": [
{
"name": "wabbit-networks-images",
"registryScopes": [ "$REGISTRY/$REPO" ],
"signatureVerification": {
"level" : "strict"
},
"trustStores": [ "$STORE_TYPE:$STORE_NAME" ],
"trustedIdentities": [
"x509.subject: $CERT_SUBJECT"
]
}
]
}
EOF
The above trustpolicy.json
file defines one trust policy named wabbit-networks-images
. This trust policy applies to all the artifacts stored in the $REGISTRY/$REPO
repositories. The named trust store $STORE_NAME
of type $STORE_TYPE
contains the root certificates. It also assumes that the user trusts a specific identity with the X.509 subject $CERT_SUBJECT
. For more details, see Trust store and trust policy specification.
Use notation policy
to import the trust policy configuration from trustpolicy.json
.
notation policy import ./trustpolicy.json
Show the trust policy configuration to confirm its successful import.
notation policy show
Use notation verify
to verify the integrity of the image:
notation verify $IMAGE
Upon successful verification of the image using the trust policy, the sha256 digest of the verified image is returned in a successful output message. An example of output:
Successfully verified signature for myregistry.azurecr.io/net-monitor@sha256:17cc5dd7dfb8739e19e33e43680e43071f07497ed716814f3ac80bd4aac1b58f
Since Notation v1.2.0 release, Notation supports RFC 3161 compliant timestamping. This enhancement extends the trust of signatures created within the certificate's validity period by trusting a Timestamping Authority (TSA), enabling successful signature verification even after the certificates have expired. As an image signer, you should ensure that you sign container images with timestamps generated by a trusted TSA. As an image verifier, to verify timestamps, you should ensure that you trust both the image signer and the associated TSA, and establish trust through trust stores and trust policies. Timestamping reduces costs by eliminating the need to periodically re-sign images due to certificate expiry, which is especially critical when using short-lived certificates. For detailed instructions on how to sign and verify using timestamping, refer to the Notary Project timestamping guide.
What should I do if the certificate is expired?
If your certificate has expired, you need to obtain a new one from a trusted CA vendor along with a new private key. An expired certificate cannot be used to sign container images. For images that were signed before the certificate expired, they may still be validated successfully if they were signed with timestamping. Without timestamping, the signature verification will fail, and you will need to re-sign those images with the new certificate for successful verification.
What should I do if the certificate is revoked?
If your certificate is revoked, it invalidates the signature. This can happen for several reasons, such as the private key being compromised or changes in the certificate holder's affiliation. To resolve this issue, you should first ensure your source code and build environment are up-to-date and secure. Then, build container images from the source code, obtain a new certificate from a trusted CA vendor along with a new private key, and sign new container images with the new certificate by following this guide.
Notation provides CI/CD solutions on Azure Pipelines and GitHub Actions:
To ensure only trusted container images are deployed on Azure Kubernetes Service (AKS):
Events
Mar 17, 11 PM - Mar 21, 11 PM
Join the meetup series to build scalable AI solutions based on real-world use cases with fellow developers and experts.
Register nowTraining
Module
Deploy and use Azure Container Registry - Training
Learn how to create a private registry service for building, storing, and managing container images and related artifacts.
Certification
Microsoft Certified: Azure Administrator Associate - Certifications
Demonstrate key skills to configure, manage, secure, and administer key professional functions in Microsoft Azure.