This reference architecture illustrates how developers can create, manage, and monitor deployed containers in the public cloud, across multiple clouds, and on-premises.
Architecture
Download a Visio file of this architecture.
Components
- Azure Container Registry is a service that creates a managed registry. It builds, stores, and manages container images and can store containerized machine learning models.
- Azure Kubernetes Service (AKS) is a managed service that offers a managed Kubernetes cluster with elastic scale-out functionality. In this architecture, a local Kubernetes cluster is used to run multiple containers on-premises.
- Azure Container Instances runs containers on-demand in a serverless Azure environment. Azure Container Instances is a low-friction method of running containers that doesn't require a full Docker host or Kubernetes installation.
- Azure Cosmos DB is a multiple model database that can serve data elastically at a massive scale. It was designed for applications that are globally distributed in a multi-write model.
- Azure Key Vault is a hardware-backed credential management service that has tight integration with Microsoft identity services and compute resources.
- Azure Policy enforces standards and assesses compliance for targeted resources that are deployed to Azure.
- Azure Private Link creates a private endpoint in your virtual network that you can use to communicate with Azure platform as a service (PaaS) without exposing your service to the public internet.
- Azure Monitor is an all-encompassing suite of monitoring services for applications that deploy in Azure or on-premises.
- Microsoft Defender for Cloud is a unified security management and threat protection system for workloads across on-premises, multiple clouds, and Azure.
Scenario details
Potential use cases
Typical uses for this architecture include:
- Web applications that have internal and external components that deploy both to the public cloud and on-premises by using shared container images.
- Modern deployment testing cycles with quality analysis, testing, development, or staging that's hosted on-premises and in the public cloud.
Recommendations
Azure Container Registry
Azure Container Registry is an enterprise container registry that can implement common best practices by protecting images from unauthorized access, replicating images across multiple geographies, preventing unnecessary ingress/egress, and optimizing costs. It supports turnkey geo-replication across multiple Azure regions, which helps you minimize latency between Azure Container Registry, your container hosts, and your development team.
Azure Container Registry includes a suite of tasks, referred to as ACR Tasks, that can manage cloud-based container image building and maintenance across a variety of operating systems. ACR Tasks can be triggered manually, by a change to source control, by a change to the base container image, or on a fixed schedule. The following are scenarios in which you could use ACR Tasks:
- An Internet of Things (IoT) developer is building container images to run on ARM-based IoT devices. The developer might be using a Linux or macOS operating system to develop the software, but needs to perform the build on an ARM platform.
- A software as a service (SaaS) development team builds software on Windows computers that run container images on Linux hosts. The team wants its builds to be done on a Linux host.
- An open source project maintainer is building a container image that augments a well-known operating system base image. The maintainer wants the container image to update every time that the base image updates.
Note
ACR Tasks can standardize the build environment and perform continuous integration of your container images.
Azure Container Instances
Azure Container Instances is a low-friction, serverless compute environment for containerized applications. It's an excellent choice for container deployment because of its low management overhead and quick startup times. Container images that are stored in Azure Container Registry can deploy directly to Azure Container Instances container groups.
In this architecture, Azure Container Instances container groups are used as virtual nodes for an Azure Kubernetes Service cluster. AKS uses virtual nodes to register a virtual pod with unlimited capacity and the ability to dispatch pods by using Azure Container Instances container groups. This is ideal when you want fast provisioning of individual pods and only want to pay for the execution time per second.
Considerations
These considerations implement the pillars of the Azure Well-Architected Framework, which is a set of guiding tenets that can be used to improve the quality of a workload. For more information, see Microsoft Azure Well-Architected Framework.
Reliability
Reliability ensures your application can meet the commitments you make to your customers. For more information, see Overview of the reliability pillar.
- Modern applications typically include a website, one or more HTTP APIs, and a connection to a data store. Applications within a container image should be stateless for maximum horizontal scale and availability. Data should be stored in a separate service that has similar availability. For guidance on designing an application that can scale to thousands of nodes, see the performance efficiency section of the Azure Well-Architected Framework.
- AKS has a reference architecture baseline that defines each of the Well-Architected Framework categories and recommends an implementation that's in line with the category.
- To reduce the impact of large pulls of container images, deploy Azure Container Registry in a region that's closest to the development team and to the production compute services. Consider a geo-replicated Azure Container Registry deployment for distributed teams and distributed production containers.
- Azure Cosmos DB is a database service that supports turnkey global distribution and supports automatic failover across multiple regions. Azure Cosmos DB also has the ability to enable multiple region writes and dynamically add or remove regions.
Security
Security provides assurances against deliberate attacks and the abuse of your valuable data and systems. For more information, see Overview of the security pillar.
Use Azure Private Link to communicate to and across services in your virtual network. By doing this, you route traffic through specific subnets to reach the individual Azure services directly and protect your data from inadvertent exposure to the public internet.
Cost optimization
Cost optimization is about looking at ways to reduce unnecessary expenses and improve operational efficiencies. For more information, see Overview of the cost optimization pillar.
- Use the Azure pricing calculator to estimate costs.
- If your development team and production instances are in a single region, consider placing the Container Registry resource in the same region. By doing this, you minimize container push and pull latency and avoid the higher costs of the Premium Azure Container Registry service tier.
- Configuring Azure Container Registry to use an Azure Virtual Network through an Azure Private Link service endpoint requires that the Azure Container Registry instance is deployed in the Premium tier.
- AKS offers free cluster management. You're only billed for the compute, storage, and networking resources that AKS uses to host nodes. See Azure Virtual Machine or Azure Container Instances pricing to review pricing for each compute service.
- If you require a specific uptime service-level agreement (SLA), you can enable the uptime SLA optional feature of AKS.
- Azure Container Instances resources are billed by the second, based on an allocation of virtual CPU and memory resources to the container group. Allocating unnecessary compute resources can significantly increase the costs of running this architecture solution. Cost monitoring and optimization is a continuous process that should be conducted at regular intervals throughout the lifetime of your deployment. For more information on minimizing Azure Container Instances operational costs, see the cost optimization section of the Azure Well-Architected Framework.
Operational Excellence
Operational excellence covers the operations processes that deploy an application and keep it running in production. For more information, see Overview of the operational excellence pillar.
Manageability
- Consider using Azure Resource Manager templates to deploy Azure Container Instance container groups in a repeatable fashion for multiple region deployments and large-scale orchestration. You can similarly use Azure Resource Manager templates to deploy Azure Kubernetes Service, Azure Key Vault, and Azure Cosmos DB.
- Consider using Azure role-based access control (Azure RBAC) to prevent users from accidentally creating or deleting container instances without permission.
- Use Azure Monitor to monitor metrics and logs for both on-premises and remote containers, analyze the data using queries, and create alerts for abnormal situations.
- Use Azure Policy to implement enforcement of a set of rules for clusters and pods that are deployed to Kubernetes Service or an Azure Arc-enabled Kubernetes cluster.
DevOps
- Use ACR Tasks to automate the build of container images on a schedule or when changes are made to the source code.
- Consider using ACR Tasks to automatically update container images as base images are patched and updated.
- The AKS team has developed GitHub actions that can assist with implementing GitOps and can facilitate deployments from Azure Container Registry to AKS clusters.
- If your Kubernetes cluster is attached to Azure Arc, you can manage your Kubernetes cluster by using GitOps. To review best practices for connecting a hybrid Kubernetes cluster to Azure Arc, see the Azure Arc hybrid management and deployment for Kubernetes clusters reference architecture.
Performance efficiency
Performance efficiency is the ability of your workload to scale to meet the demands placed on it by users in an efficient manner. For more information, see Performance efficiency pillar overview.
- Customer-facing containerized web applications benefit from variable scales. You can use services such as Azure Container Instances and AKS to dynamically scale out to meet anticipated or measured demand. You can also use services such as Azure Functions and Azure App Service to run container images at scale.
- Internal application usage is more predictable and can run on an existing Kubernetes cluster. If you're interested in deploying Azure-managed services on-premises, consider:
- Azure Cosmos DB automatically scales service resources to meet the storage needs of your application in an elastic manner. For throughput, you can choose to pre-provision throughput or operate Azure Cosmos DB as a serverless service. If your workload has variable or unpredictable demands, you can also choose to provision your throughput using autoscale.
- Modern applications typically include a website, one or more HTTP APIs, and a connection to a data store. Applications within a container image should be stateless for maximum horizontal scale and availability. Data should be stored in a separate service that has similar availability. For guidance on designing an application that can scale to thousands of nodes, see the performance efficiency section of the Azure Well-Architected Framework.
- AKS has a reference architecture baseline that defines each of the Well-Architected Framework categories and recommends an implementation that's in line with the category.
- To reduce the impact of large pulls of container images, deploy Azure Container Registry in a region that's closest to the development team and to the production compute services. Consider a geo-replicated Azure Container Registry deployment for distributed teams and distributed production containers.
- Azure Cosmos DB is a database service that supports turnkey global distribution and supports automatic failover across multiple regions. Azure Cosmos DB also has the ability to enable multiple region writes and dynamically add or remove regions.
Contributors
This article is maintained by Microsoft. It was originally written by the following contributors.
Principal author:
- Pieter de Bruin | Senior Program Manager
To see non-public LinkedIn profiles, sign in to LinkedIn.
Next steps
- Azure Container Registry documentation
- Azure Kubernetes Service documentation
- Azure Policy documentation
- Azure Monitor documentation
- Azure Container Instances documentation
- Azure Cosmos DB documentation
- Azure Key Vault documentation
- Azure Private Link documentation
- Microsoft Defender for Cloud documentation
- Kubernetes documentation
Related resources
Related hybrid guidance:
- Hybrid architecture design
- Azure hybrid options
- Hybrid app design considerations
- Deploy a hybrid app with on-premises data that scales cross-cloud
Related architectures:
- Enterprise infrastructure as code using Bicep and Azure Container Registry
- Baseline architecture for an Azure Kubernetes Service (AKS) cluster
- Microservices architecture on Azure Kubernetes Service
- Advanced Azure Kubernetes Service (AKS) microservices architecture
- GitOps for Azure Kubernetes Service
- Monitor a microservices architecture in Azure Kubernetes Service (AKS)
- Enterprise monitoring with Azure Monitor