Prepare Azure container technical assets for a Kubernetes app
This article gives technical resources and recommendations to help you create a container offer on Azure Marketplace for a Kubernetes application.
For a comprehensive example of the technical assets required for a Kubernetes app-based Container offer, see Azure Marketplace Container offer samples for Kubernetes.
Fundamental technical knowledge
Designing, building, and testing these assets takes time and requires technical knowledge of both the Azure platform and the technologies used to build the offer.
In addition to your solution domain, your engineering team should have knowledge about the following Microsoft technologies:
- Basic understanding of Azure Services
- How to design and architect Azure applications
- Working knowledge of Azure Resource Manager
- Working knowledge of JSON
- Working knowledge of Helm
- Working knowledge of createUiDefinition
- Working knowledge of Azure Resource Manager (ARM) templates
Your application must be Helm chart-based.
All the image references and digest details must be included in the chart. No additional charts or images can be downloaded at runtime.
You must have an active publishing tenant or access to a publishing tenant and Partner Center account.
You must have created an Azure Container Registry (ACR) to which you'll upload the Cloud Native Application Bundle (CNAB), and give permission to Microsoft’s first party app ID to access your ACR. For more information, see create an Azure Container Registry.
Install the latest version of the Azure CLI.
The application must be deployable to Linux environment.
If running the CNAB packaging tool manually, you will need docker installed on your local machine.
- Container Marketplace supports only Linux platform-based AMD64 images.
- Managed AKS only.
- Single containers are not supported.
- Linked Azure Resource Manager templates are not supported.
The Kubernetes application-based offer experience is in preview. Preview features are available on a self-service, opt-in basis. Previews are provided "as is" and "as available," and they're excluded from the service-level agreements and limited warranty. Previews are partially covered by customer support on a best-effort basis. As such, these features aren't meant for production use.
The first step to publish your Kubernetes app-based Container offer on the Azure Marketplace is to package your application as a Cloud Native Application Bundle (CNAB). This CNAB, comprised of your application’s artifacts, will be first published to your private Azure Container Registry (ACR) and later pushed to a Microsoft-owned ACR and will be used as the single artifact you reference in Partner Center.
From there, vulnerability scanning is performed to ensure images are secure. Finally, the Kubernetes application is registered as an extension type for an Azure Kubernetes Service (AKS) cluster.
Once your offer is published, your application will leverage the cluster extensions for AKS feature to manage your application lifecycle inside an AKS cluster.
Grant access to your Azure Container Registry
As part of the publishing process, Microsoft will deep copy your CNAB from your ACR to a Microsoft-owned, Azure Marketplace-specific ACR. This step requires you to grant Microsoft access to your registry.
Microsoft has created a first-party application responsible for handling this process with an
32597670-3e15-4def-8851-614ff48c1efa. To begin, create a service principal based off of the application:
If your account doesn't have permission to create a service principal,
az ad sp create will return an error message containing "Insufficient privileges to complete the operation". Contact your Azure Active Directory admin to create a service principal.
az login az ad sp create --id 32597670-3e15-4def-8851-614ff48c1efa
Make note of the service principal's ID to use in the following steps.
Next, obtain your registry's full ID:
az acr show --name <registry-name> --query "id" --output tsv
Your output should look similar to the following:
... }, "id": "/subscriptions/ffffffff-ff6d-ff22-77ff-ffffffffffff/resourceGroups/myResourceGroup/providers/Microsoft.ContainerRegistry/registries/myregistry", ...
Next, create a role assignment to grant the service principal the ability to pull from your registry using the values you obtained earlier:
To assign Azure roles, you must have:
Microsoft.Authorization/roleAssignments/writepermissions, such as User Access Administrator or Owner
az role assignment create --assignee <sp-id> --scope <registry-id> --role acrpull
Finally, register the
Microsoft.PartnerCenterIngestion resource provider on the same subscription used to create the Azure Container Registry:
az provider register --namespace Microsoft.PartnerCenterIngestion --subscription <subscription-id> --wait
Monitor the registration and confirm it has completed before proceeding:
az provider show -n Microsoft.PartnerCenterIngestion --subscription <subscription-id>
Gather artifacts to meet the package format requirements
Each CNAB will be composed of the following artifacts:
Update the Helm chart
Ensure the Helm chart adheres to the following rules:
All image names and references are parameterized and represented in
values.yamlas global.azure.images references. Update
deployment.yamlto point these images. This ensures the image block can be updated and referenced by Azure Marketplace's ACR.
If you have any subcharts, extract the content under charts and update each of your dependent image references to point to the images included in the main chart's
Images must use digests instead of tags. This ensures CNAB building is deterministic.
Available billing models
|Licensing option||Transaction process|
|Free||List your offer to customers for free.|
|Bring your own licensing (BYOL)||The Bring Your Own Licensing option lets your customers bring existing software licenses to Azure.*|
|Per every core in cluster||List your Azure Container offer with pricing charged based on the total number of CPU cores in the cluster (reported at hourly frequency). You provide the price for one CPU core and we’ll increment the pricing based on the total number of CPU cores in the cluster.|
|Per core||List your Azure Container offer with pricing charged for every core used by the Kubernetes application’s extension instance (reported at hourly frequency). You provide the price for one CPU core and we’ll increment the pricing based on the cores used by the Kubernetes application instance in the cluster.|
|Per cluster||List your Azure Container offer with pricing charged for each instance of the Kubernetes application extension on the cluster (reported at hourly frequency). You provide the price for one instance of the Kubernetes application and we’ll increment the pricing based on the number of instances of the Kubernetes application on the cluster.|
|Per every node in cluster||List your Azure Container offer with pricing charged based on total number of nodes in the cluster (reported at hourly frequency). You provide the price for one node in the cluster and we’ll increment the pricing based on the size of hardware in the cluster.|
|Per node||List your Azure Container offer with pricing charged for each node on which the Kubernetes application’s extension instance runs on (reported at hourly frequency). You provide the price for one node in the cluster and we’ll increment the pricing based on the number of nodes on which the Kubernetes application instance runs in the cluster.|
|Per pod||List your Azure Container offer with pricing charged for each pod on which the Kubernetes application’s extension instance runs (reported at hourly frequency). You provide the price for one node in the cluster and we’ll increment the pricing based on the number of pods used on which the Kubernetes application instance runs in the cluster.|
- As the publisher, you support all aspects of the software license transaction, including (but not limited to) order, fulfillment, metering, billing, invoicing, payment, and collection.
Make updates based on your billing model
After reviewing the billing models available, select one appropriate for your use case and complete the following steps:
Complete the following steps to add identifier in the Per core billing model:
Add a billing identifier label and cpu cores request to your
Add a billing identifier value for
Complete the following steps to add a billing identifier label in the Per pod and Per node billing model:
- Add a billing identifier label
deployment.yamlfile (Under Template > Metadata > Labels>).
Note that at deployment time, the cluster extensions feature will replace the billing identifier value with the extension type name you provide while setting up plan details.
For examples configured to deploy the Azure Voting App, see the following:
Validate the Helm chart
To ensure the Helm chart is valid, test that it's installable on a local cluster. You can also use
helm install --generate-name --dry-run --debug to detect certain template generation errors.
Create and test the createUiDefinition
A createUiDefinition is a JSON file that defines the user interface elements for the Azure portal when deploying the application. For more information, see CreateUiDefinition.json for Azure or see an example of a UI definition that asks for input data for a new or existing cluster choice and passes parameters into your application.
After creating the createUiDefinition.json file for your application, you need to test the user experience. To simplify testing, use a sandbox environment that loads your file in the portal. The sandbox presents your user interface in the current, full-screen portal experience. The sandbox is the recommended way to preview the user interface.
Create the Azure Resource Manager (ARM) template
An ARM template defines the Azure resources to deploy. You will be deploying a cluster extension resource for the Azure Marketplace application. Optionally, you can choose to deploy an AKS cluster.
We currently only allow the following resource types:
For example, see this sample ARM template designed to take results from the sample UI definition linked above and pass parameters into your application.
Create the manifest file
The package manifest is a yaml file that describes the package and its contents, and tells the packaging tool where to locate the dependent artifacts.
The fields used in the manifest are as follows:
|applicationName||String||Name of the application|
|publisher||String||Name of the Publisher|
|description||String||Short description of the package|
||Version string that describes the application package version, may or may not match the version of the binaries inside. Mapped to Porter’s version field|
|helmChart||String||Local directory where the Helm chart can be found relative to this
|clusterARMTemplate||String||Local path where an ARM template that describes an AKS cluster that meets the requirements in restrictions field can be found|
|uiDefinition||String||Local path where a JSON file that describes an Azure portal Create experience can be found|
|registryServer||String||The ACR where the final CNAB bundle should be pushed|
|extensionRegistrationParameters||Collection||Specification for the extension registration parameters. Include at least
|defaultScope||String||The default scope for your extension installation. Accepted values are
|namespace||String||(Optional) Specify the namespace the extension will install into. This property is required when
For a sample configured for the voting app, see the following manifest file example.
User parameter flow
It's important to understand how user parameters flow throughout the artifacts you're creating and packaging. Parameters are initially defined when creating the UI through a createUiDefinition.json file:
In this example,
extensionResourceName is also parameterized and passed to the cluster extension resource. Similarly, other extension properties can be parameterized, such as enabling auto upgrade for minor versions. For more on cluster extension properties, see optional parameters.
and are exported via the
From there, the values are passed to the Azure Resource Manager template and will be propagated to the Helm chart during deployment:
Finally, the values are consumed by the Helm chart:
Structure your application
Place the createUiDefinition, ARM template, and manifest file beside your application's Helm chart.
For an example of a properly structured directory, see the sample repository.
Use the container packaging tool
Once you've added all the required artifacts, run the packaging tool
Since CNABs are a generally new format and have a learning curve, we've created a Docker image for
container-package-app with bootstrapping environment and tools required to successfully run the packaging tool.
You have two options to use the packaging tool. You can use it manually or integrate it into a deployment pipeline.
Manually run the packaging tool
The latest image of the packaging tool can be pulled from
The following Docker command pulls the latest packaging tool image and also mounts a directory.
~\<path-to-content> is a directory containing the contents to be packaged, the following docker command will mount
/data in the container. Be sure to replace
~/<path-to-content> with your own app's location.
docker pull mcr.microsoft.com/container-package-app:latest docker run -it -v /var/run/docker.sock:/var/run/docker.sock -v ~/<path-to-content>:/data --entrypoint "/bin/bash" mcr.microsoft.com/container-package-app:latest
Run the following commands in the
container-package-app container shell. Be sure to replace
<registry-name> with the name of your ACR:
export REGISTRY_NAME=<registry-name> az login az acr login -n $REGISTRY_NAME cd /data/<path-to-content>
cpa verify to iterate through the artifacts and validate them one by one. Address any failures, and run
cpa buildbundle when you're ready to package and upload the CNAB to your Azure Container Registry. The
cpa buildbundle command will also run the verification process before building.
cpa verify cpa buildbundle
cpa buildbundle --force only if you want to overwrite an existing tag. If you have already attach this CNAB to an Azure Marketplace offer, instead increment the version in the manifest file.
Integrate into an Azure Pipeline
For an example of how to integrate
container-package-app into an Azure Pipeline, see the Azure Pipeline example.