Steve Lasker's Web Log - https://SteveLasker.blog

Moving the cheese - SteveLasker.blog

Steve Lasker — Mon, 31 Dec 2018 01:30:25 GMT

Microsoft is retiring personal MSDN blogs. As a result, I've moved all my blog content to: https://SteveLasker.blog I hope you'll keep the great feedback coming there. Steve

ACR Build (Preview) Now Supported In All Public Regions

Steve Lasker — Wed, 25 Jul 2018 22:15:23 GMT

ACR Build, a native container build capability of the Azure Container Registry is now available in all regions ACR is available. That's 23 regions worldwide as of July 23rd 2018. ACR Build enables network close builds, alongside your container registry. While docker has an efficient image layering scheme, image layers can grow in size. Reducing the network distance your image pull and push operations must travel is key to improving reliability, speed and reducing network egress costs. Wherever you run your Azure Container Registry, you can now build Linux, Windows and ARM right alongside.

OS & Framework Patching of Containers

ACR Build is primarily focused on the container lifecycle management challenge. ACR Build supports:

builds from your local directory az acr build .
git commit based builds az acr build-task create
base image updates, providing Container OS & Framework patching

ACR Build is currently in preview, incorporating feedback. Based on feedback, we've recently enabled:

remote context, such as an existing git repository with nested folders az acr build \ -t hello-acr:{{.Build.ID}} \ https://github.com/Azure-Samples/acr-helloworld.git and nested folders az acr build \ -t hello-world-of-windows:{{.Build.ID}} \ https://github.com/AzureCR/acr-builder-os-tests.git#master:Windows/Servercore
multi-tagging support -t image1:{{.Build.ID}} -t image1:latest
ARM support
Windows support
{{.Build.Date}} is coming soon

Watch more on Azure Friday with Lara Rubbelke Give it a try. Tell us what you need.

For more information

Any other feedback, feel free to leave larger comments here, user voice, twitter (@SteveLasker) Thanks, Steve and the great members of the ACR team

Azure Container Registry Build Supports All Windows Versions (Preview of Preview)

Steve Lasker — Mon, 16 Jul 2018 23:00:50 GMT

ACR Build, a cloud-native container build capability of the Azure Container Registry now supports all supported versions of Windows Containers. In May of 2018, Azure announced ACR Build (Preview), a component of Azure's OS & Framework Container Life-cycle Management Today, we wanted to give customers early access to Windows Builds, in addition to the Linux and ARM builds customers could do since last May.

ACR Build Enables All Supported Versions of Windows Containers

The most significant element of Windows support is the support for all supported version of Windows, including ltsc2016, 1709 and 1803. In addition to Windows Server Core, Nano Server is also supported by ACR Build. A suite of tests for Windows containers are available here: https://github.com/AzureCR/acr-builder-os-tests/tree/master/Windows

What is (Preview of Preview)

Updates to the az cli will not be publicly available until July 31. There are two ways to get ACR Build for Windows Container support today (July 16th, 2018)

Create a build task with the public az cli build-task already supports the --os parameter, so you're good to go if you just want build-tasks
To use az acr build; the equivalent of docker build, use a preview build of the az cli, which can be
1. installed directly: https://github.com/Azure/azure-cli#edge-builds
2. or with a docker image: https://github.com/Azure/azure-cli#docker docker run -v ${HOME}:/root -it azuresdk/azure-cli-python:dev

A quick test

We'll use the most consistent way, a docker container of the dev builds of the az cli

On Windows, or Mac, instance a docker container, running the latest dev build, including updates to az acr build docker run -v ${HOME}:/root -it azuresdk/azure-cli-python:dev
Note: ACR Build is currently supported:
- East US
- West Europe
- Central US
- South Central US More regions are coming on weekly, with full global coverage coming "soon"
Build a Windows Server Core image, with a .NET Framework app az acr build \ -t helloworld-windowsservercore:multi-arch-{{.Build.ID}} \ -f helloworld-windowsservercore/multi-arch.Dockerfile \ --os windows \ https://github.com/AzureCR/acr-builder-os-tests.git#master:Windows/Servercore

How this works:

az acr build is the functional equivalent of docker build. Rather than having to install the docker client, az acr build sends the context to an ACR Build server in Azure
-t uses the standard docker tag, with the added ability to get a unique build id.
-f uses the standard dockerfile syntax to reference a Dockerfile that isn't in the root of the context
--os tells ACR Build to use a Windows host. As many FX and platform tags are multi-arch, this parameter is required to specify which OS you require for docker build
https://github... is a remote context for the build. While . is supported for a local directory, acr build and build-task can also specify remote git repos, avoiding the necessity of first git clone

Since this references the multi-arch tag for Windows and the .NET Framework, this will produce an 1803 version of Windows Server Core. To get a specific windows version, use the following:

Windows Server 1709, using the 1709.Dockerfile az acr build \ -t helloworld-windowsservercore:1709-{{.Build.ID}} \ -f helloworld-windowsservercore/1709.Dockerfile \ --os windows \ https://github.com/AzureCR/acr-builder-os-tests.git#master:Windows/Servercore
Windows Nano Server 1709, using the 1709.Dockerfile for Nano Server and .NET Core az acr build \ -t helloworld-nanoserver:1709-{{.Build.ID}} \ -f helloworld-nanoserver/1709.Dockerfile \ --os windows \ https://github.com/AzureCR/acr-builder-os-tests.git#master:Windows/Nanoserver

Automated Build Tasks, based on Git Commits, or Based Image Updates

The primary scenario for ACR Build revolves around Container OS & Framework patching. Just as your builds kick off, based on git-commit changes, as base images are patched, your builds can also be triggered. See Overview of ACR Build, targeting OS & Framework Patching for more details.

Creating automated builds

Note: this can be done directly with the publicly released az cli, however, we'll use the az cli docker image for consistency.

Fork the https://github.com/AzureCR/acr-builder-os-tests repo
Create a Personal Access Token. For more info, see the ACR Build Docs for creating a build task.
Set the PAT to an environment variable PAT=[pastePAT]
Create an acr build-task az acr build-task create \ -n HelloworldWinServercoreLtsc2016 \ -t helloworld-windowsservercore:ltsc2016-{{.Build.ID}} \ -f helloworld-windowsservercore/ltsc2016.Dockerfile \ -c https://github.com/AzureCR/acr-builder-os-tests.git#master:Windows/Servercore \ --os windows \ --git-access-token $PAT
Trigger a build manually az acr build-task run HelloworldWinServercoreLtsc2016

For more information

Any other feedback, feel free to leave larger comments here, user voice, twitter (@SteveLasker) Thanks, Steve and the great members of the ACR team

Key Differences between VM and Container Vulnerability Scanning

Steve Lasker — Wed, 27 Jun 2018 20:22:10 GMT

As we explore the various options for integrating vulnerability scanning into containers running in Azure, we encountered many different approaches and requests. I was initially writing a document for internal discussions, outlining some of our internal goals for how we want to think about container security, and why we might take one approach or another. While I'd suggest Azure has the best breadth and depth of containers in any of the major clouds, we are a large company, and not everyone is familiar with containers. In our busy lives, we only have a few seconds of mental processing, before we look to apply what we've just learned to the problems we have. Containers solve many problems, some more short term than others. However, until you've inverted your thought process, and think about a container-first mindset, you'll find yourself using a new tech as a better tool, but not nearly unlocking its potential. As I wrote the document, there are lots of details and examples I wanted to outline to help people think about vulnerability from a container first mindset. Many of which represent opinions, or "best practices". As I wrote the Docker tagging best practices for tagging and versioning docker-images post, I thought about what makes a best practice for a new and rapidly evolving technology? I don't consider myself a security expert, and I certainly don't play one on Twitter. I have worked in the industry for many years and have watched the balance of secure, usable and practical. We don't always get these right. And I don't believe the concept that something must be hard to be secure. I believe in good, obvious processes and realistic expectations for making something secure. After all, what's the first thing developers do when security gets in the way? ...we disable it. Not a very secure result, now is it? To put some context to this paradigm shift, below are a number of concepts to consider when you think about adopting a container scanning solution. Just because something has been done a certain way for years, doesn't mean it's still relevant.

Virtual Machine and Container Differences

Below are several concepts we think about when running compute. I've provided a VM centric and Container centric view to each one.

Container Bigot Disclaimer: For those working with products like Vagrant, Terraform and other awesome products that automate VM creation; this isn't meant to suggest the challenges couldn't/aren't addressed with them. And while I'm far to say anything you can do in a VM, I can do better in a container, I think we all recognize containers, and specifically Docker has brought about the next foundational shift in how we package and run software.

Where Is Fast a Priority

VM VMs are typically measured in the length of up-time. How long was the VM running before it had to be rebooted or retired? While startup time is always important, VM startup time is typically measured in minutes. As long as they come up in single digits, they’re considered “the norm”. Container Containers are assumed to be relatively short-lived processes. They may be long running services, like websites or APIs, but the expectation is they get moved, updated, scaled with far more frequency. Containers are measured in the number of seconds, or preferably milliseconds from the time of docker run to processing requests. Customers are also looking for docker pull to complete in seconds, reducing the amount of time to run a new image on a new node. Considering containers are becoming the base of functions, where a workflow kicks off, and each step of the process may be its own container that’s pulled, instanced and tossed in the matter of seconds, anything that can slow down the instancing must be changed.

Configuring and Applying Changes

VM While projects/products like Terraform, Spinnaker, Vagrant and others aim to automate the final configuration of VMs, they are most commonly configured post instancing. Teams routinely remote into the machine, applying software or patches. Even when it's not intended, many teams still do, and these long running nodes need to be protected. Container Containers are considered immutable. Meaning, once an image is instanced, it’s not changed. One would likely argue a running container shouldn’t be changed as it defeats the ability for a new instance to reflect the exact behavior. Rather than update a running container, the image dockerfile is changed to implement the configuration change or software. The image is rebuilt, pushed to a registry, and awaits its time to be instanced.

Applying OS & Framework Patching

VM Like configuring and applying changes, teams work to update templates for deployments. However, most operations teams will monitor and update their running VMs. These are updated, without necessarily understanding what apps are running, as it's just too difficult for companies to track which VMs could be impacted. Container Also, like configuring and applying changes, a container isn’t patched while running. The image is rebuilt with the OS and framework patches. In this model, the image can be scanned and tested before its deployed.

Receiving Data

VM As VMs are designed as long running instances, VMs are designed to host data. It may involve downloading EDI files to be processed, customer binaries to signed, pictures to be processed. In each case, the VM has the ability to download malicious content. If the VM is found to be vulnerable, the data on the VM must be scrubbed and typically recovered. Container Containers are considered immutable processes that may be run for seconds or hours. Any data that may be placed in a container would/should be considered disposable. The container must be assumed to fail at any point. Containers store their data externally, using volume drivers or data services like DocDB, Relational or Blob storage. An image that becomes suspect could/should be deleted without any worry for what temporary storage that may have been written.

Multiple Lines of Defense

VM In the VM world, the VM itself has multiple lines of defense. It may be secured within a VNNet, locked down to where it may communicate through firewalls, which also run on the VM. Scanning is placed within each running VM. Container In the container world, securing the process within the container is the last line of defense. Containers themselves do not actively run scanning. They are lightweight processes. The container host may likely run scanning for what images are being deployed. In the best world, the image being requested to run is verified against a list of pre-scanned images. If the image has been already, and recently scanned, it’s allowed to run. The first line of defense for containers is the source where they are kept, before deployment. This is referred to as the container registry. Each company running containers will host their own private registry. The scanner keeps an inventory of all images in the company’s private registries. Scanning at the source allows offline scanning, before the image has been requested to be deployed. This makes instancing a container fast as it’s just verified to have been seen prior. The host monitoring will either block or scan images it hasn’t previously seen.

Vulnerability Detection and Impact

VM In the VM world, for an enterprise solution, scan results are reported back to a central server. However, the report indicates what was found and the action. It doesn’t necessarily trigger a recursive action for other like VMs. Container In the container world, the registry is actively scanned. Each protected node may report back what images it’s running. As the scanner finds existing vulnerabilities in newly pushed images to the container registry, or it finds new vulnerabilities in existing images in the registry, the scanner determines the impact. It can use the data for what images are running (referenced by their image digest) to understand the impact of the vulnerability. If the scanner happens to find a vulnerability that was in a single image that was never deployed, the risk is non-impactful. The action is as simple as either quarantining the image, or deleting it. If the scanner finds a new, critical vulnerability in a base image that has 50 other derived images, which are deployed on 220 production nodes, the impact is far more severe. The fundamental difference between containers and VMs is where the scanning occurs, with the analysis and action taken. In the container world, a single scan and detection can account for hundreds to thousands of nodes being remediated.

Remediating Vulnerabilities

VM Remediating vulnerabilities in a VM typically involves applying a patch to a running VM. The intent involves the patch removing or replacing the initial offending binary or configuration. Container When vulnerabilities are found with containers, a new, remediated image is built, pushed and deployed. Some of the key differences being the original configuration and/or binary shouldn’t have any traits in the newly built image. When the vulnerability is found in a top-level image, meaning an image that copied the offending binary in its dockerfile, it’s built once and deployed to the intended nodes. To remediate a base image, all the subsequently derived images must be rebuilt. A shameless plug for ACR Build. In the VM environment, without products like Terraform, each node must be individually updated.

The Scanning Paradigm Shift

With the above differences, we can see the approach for scanning containers would be very different from Virtual Machines. The basic approach of containers represents a paradigm shift from working with virtual machines. While Virtual Machines were a major change in 2001, improving many challenges of installations of the OS and software, installed on dedicated hardware, they are now nearly two decades old. We’ve learned a lot since. We’ve changed the workloads from machines under someone’s desk, in a closet with hopefully enough air conditioning, or a bigger closet in the basement of an enterprise. Expectations of 3 months to get infrastructure acquired and provisioned are long gone. Customers spin up instances in an instant, on-demand, with expectations of seconds in a cloud they have no care as to where it runs. Containers aren’t just a better mousetrap, they account for many of the lessons the industry has learned and adjusted to as the workloads have grown and transitioned from assisting humans at work. Compute now accounts for most of the work, and the humans control and interact with the computers. As a result, it’s no longer just an annoyance when the excel spreadsheet is locked or not available. Having a system down, just for a few minutes, can kill the reliability of the business, cost them millions of dollars in fines or lost revenue, or cause complete failure of the business. As with any new tech, taking the time to digest its impact, understanding the opportunity for things it can resolve, and things it can't are just as important as understanding the calorie count of your favorite new dessert.

ACR Build OS & Framework Patching Now Supports Docker Hub

Steve Lasker — Mon, 11 Jun 2018 20:33:59 GMT

In May we announced the public preview of ACR Build, enabling OS & Framework patching. When we announced the preview, only Azure Container Registries were supported for base image update notifications. With DockerCon starting this week, it's only appropriate we add support for base in Docker Hub.

Docker build a locally scoped directory in Azure, with a tag based on the build id:

git pull https://github.com/demo42/helloworld az acr build -t helloworld:{{.Build.ID}} .

Create a build-task that automatically builds, based on git commits, or base image updates

az acr build-task create \ -n helloworld \ -c https://github.com/demo42/helloworld \ -t helloworld:{{.Build.ID}} \ --git-access-token $PAT When az acr build completes, the base images dependencies are dynamically discovered. We believe dynamic discovery is key, so your build task and dependencies are kept in sync with your code:

ACR Builder discovered the following dependencies:
- image:
    registry: demo42.azurecr.io
    repository: helloworld
    tag: aanc
    digest: sha256:581a6215cae409690a4106fb06f8a1f7b9cd4790b956cb197b22eb4cae9b3977
  runtime-dependency:
    registry: registry.hub.docker.io
    repository: baseimages/node
    tag: 9-alpine
    digest: sha256:1c3d74a201d4169a2dff0d49e831510384063df7546ee89bb806febfe29eaadb
  git:
    git-head-revision: e578d0e125be1599fb125bdd07c05f16d45ff1c1

Build Latency

To meet the scale of builds that we expect over time, ACR build has a built in randomness to the delay by which a base image update will be notified. This is currently between 10 and 60 minutes. Using az acr build-task list-builds, you can see the build change from a Git Commit to Image Update

BUILD ID   TASK               PLATFORM   STATUS    TRIGGER      STARTED              DURATION
---------- ------------------ ---------- --------- ------------ -------------------- ---------
aanc       helloworld         Linux      Succeeded Image Update 2018-06-11T18:34:55Z 00:00:45
aanb       helloworld         Linux      Succeeded Git Commit   2018-06-11T18:31:13Z 00:00:45

We believe automated OS & Framework patching is a key element of container life cycle management. We also know that running ahead means we may not always have the the experience just right. We encourage your thoughts and feedback:

Getting started with ACR Build: aka.ms/acr/build
Issues with ACR Build: aka.ms/acr/issues
ACR Backlog: aka.ms/acr/backlog
Feature suggestions for ACR: aka.ms/acr/uservoice

You can always reach me directly at Steve.Lasker at microsoft dot com or twitter: @SteveLasker Steve

How ACR Build Id's are generated

Steve Lasker — Fri, 18 May 2018 19:42:38 GMT

As you use ACR Build, for OS & Framework patching, native container builds, or validating a docker build without having the docker client installed, you may be wondering, what is the format for those alpha-numeric Build id's. aa12 The short answer is it's based on the region the build was executed upon, and a base 32, alpha-numeric sequence to provide independent execution, with multi-region fault tolerance. The format is: [2 digit, base 32 region code][base 32 sequence] For some background why we chose this solution...

Accepting failure as reality

As we build ACR, and its associated services, we accept the premise that "failure is reality". One of the base foundations of container orchestrators is to accept failure as reality, at the infrastructure level. Its not that we don't strive to make ACR and all our container services reliable. But attempting to make a service n9's reliable is a costly investment. Rather than spend 90% of our effort, getting the last n% of reliability, we invest in recovering from and accepting failure, and providing more capabilities. The end result is you achieve n9's, but you do so by accepting any number of elements may individually fail, and we avoid single points of failure.

ACR Build, Independent Execution, Multi-Region Reliability

ACR supports geo-replicated registries. Which means a single registry can span multiple regions. ACR supports multi-master changes, so any push, any delete, any meta-data update to any region is eventually consistent. Meaning, there's no guarantee for the ordering of each regions replicated arrival. However, all replicated regions will eventually have all the images and meta-data. There's no single master controller. This means any registry can go down, and the rest will continue to operate. As a practice, we have no single points of failure, other than the big rock we all reside. We have planetary replication, to replicate to Mars and other destinations on our backlog, so the sky is not the limit. Each replicated registry should be able to build, on its own, and become eventually consistent.

Balancing Needs for a Unique ID

With a little background, we can see why we didn't just use a Count Dracula approach of 1, 2, 3... We then had to decide; how can we generate a unique id? We came up with a few guiding principles.

It must be unique, as we want to support Best practices for tagging and versioning docker images
It must be short enough to be able remember as you tell someone across the room, or type into another device while looking at another screen
It must be easy for a human to read, and not get confused between 1 and l. 0 and o. 2 and z.
It would be nice if there was some sense of sequential ordering. Although it wasn't critical to use every digit.
It didn't have to be globally unique. It could be unique to a specific registry. Meaning myregistry.azurecr.io and yourregistry.azurecr.io could have the same build id. Making it globally unique would conflict with principals 2-4.
We could seed each region with it's own pre-qualifier, so each geo-replicated region of a registry could be sequential

We considered prefacing the region specific unique number with the region id. eastus-1234. Although the first private preview used this, it seemed long and could be confusing when you consider that ACR Build could load balance builds across geo-replicated regions. By using generic numbers, there's no guarantee the preface characters really meant a specific thing.

Where we landed:

To keep it short, human readable, we went with a base 32 alpha-numeric sequence. All digits will be lower case, including the numbers. (what's the point of doing this if we can't laugh at the details) 0123456789abcdefghjkmnpqrstuvwxy
Each region will get a sequential two digit preface. Rather than do an elaborate zip code model, we'll take a simplistic agile approach. Every region we'll add, will simply get the next sequential two digit seed. An insider secret. You'll know the order we roll out build regions by its preface characters.

Region	Seed
East US	aa
West Europe	ab
The next region	ac

Each region will track its next id.
If a build fails, the build id is associated with the failed build, and the next sequential number is used for the next build.
Quick builds az acr build -t ., will also get sequential build ids.
Build Ids are assigned as the build is queued. If two builds are queued at the same time, that started at 1pm, that took 2 minutes will have a lower number than the build that started at 1:01 pm, and only took 30 seconds.
The preface characters are not tied to the closest registry from where the command is executed. As customers adopt base image updates, we expect huge bursts of automated builds as popular OS and run times update. As we scale ACR Build, we will stage the events, globally, just as Windows Update does today. If a region is at capacity, we may utilize any geo-replicated regions of a registry to load balance builds. If a registry is replicated between East US and West Europe, a build initiated on the east coast, may be bounced to West Europe based on capacity. ACR Geo-replication will kick in, and the Build ID will start with ab, the seed from West Europe.

We're excited to participate in building reliable systems, based on the reality that failure is a thing. We also strive on building systems that are easy to maintain as all too often, failures are triggered by complexity of simple things: quick-6-six-unit-conversion-disasters Steve

Docker Tagging: Best practices for tagging and versioning docker images

Steve Lasker — Thu, 01 Mar 2018 03:54:34 GMT

In any new tech, there are lots of thoughts around “best practices”. When a tech is new, what makes a best practice? Working at Microsoft, running the Azure Container Registry (ACR), talking with lots of customers, some that use Azure and some that don’t, we’ve had a lot of exposure to what customers have encountered. We’ve been working on a number of scenarios, including container life cycle management and OS & Framework patching with containers. This has surfaced a number of interesting issues. Issues that suggest, “both sides” of the tagging argument have value. It’s easy to get caught up, arguing a particular best practice, …if you don’t first scope what you’re practicing. I'll start by outlining two basic versioning schemes we’ve found map to the most common scenarios. And, how they are used together to solve the container life cycle management problem.

Container Patterns Support a Self-Healing Model

Before I get into specific tagging schemes, it may help to identify a specific scenario impacted by your tagging scheme. The most basic container scenario supports restart type policies. docker run --restart always myunstableimage:v0 This helps when a container fails due to some internal reasoning. But, the host is still available. In this case, our code must be resilient. Or, to an old friends point (Pat Helland), "we must write our software with a apology based computing approach." Apology based computing suggest our code must deal with "stuff" that happens. A set of code, or a container may fail and need to re-run that logic. For the purposes of tagging, I'll stay focused on the other aspect; we must assume a container host may fail at any point. Orchestrators will re-provision the failed node, requiring a new docker pull. Which begs the question: when you have a collection of containers running the same workload, do you want them to all run the same workload, or some of them have the last version, while new nodes have a newer version? In the above diagram, we have:

a local build of 3 images: web:1, api:1, cache:1. Looking closely, we can see every tag has an ID, otherwise known as a digest.
When the images are pushed to a registry, they are stored as two parts. The image:tag, and the underlying digest. These are actually separate elements, with a pointer between them.
A deployment is made, using the tag. Not the digest. Although, that is an option. Digests aren't as clean as I show here. They actually look like this: f3c98bff484e830941eed4792b27f446d8aabec8b8fb75b158a9b8324e2da252. Not something I can easily fit in this diagram, and certainly not something you'd want to type, or even try and visually find when looking at the containers running in your environment.
When a deployment is made to a host, the orchestrator keeps track of what you asked it to deploy and keep deployed. This is the important part. ...wait for it...
With a deployment saved, the orchestrator starts its work. It evaluates what nodes it has to work with and distributes the request.

A few things here. While many would argue Kubernetes has become the defacto standard for container orchestrators, orchestration is a generalized concept. Over time, we expect orchestration to just become the underpinnings to higher level services. But, that's another post...

As our development team moves along, a new version of the api and cache are built and pushed. Since this is just a minor fix to version 1, the team uses the same tag.
Our registry acknowledges the request, saving the new :1 version of the api and cache. Notice the original tags are now pointing to the new digests (3rp, 1n4)

At this point, you likely see where this is going.

As Pat would say "Stuff Happens". Actually, Pat would say something else. Our Host-B node has failed.
No problem, our orchestrator's job is to maintain a requested deployment. It sees the failure, evaluates the request made of it, and provisions the necessary resources to recover. Our deployment chart clearly says, keep 3 copies of web:1, 3 copies of api:1 and 4 copies of cache:1. So, it does a docker run of each of those image:tag combinations. It's doing what it was told. Our Host-C now has the replacement images running

Is this the desired state you want? The orchestrator did it's job. It was told to deploy and maintain a given set of images. The problem was actually further upstream, when we re-used the same tag for our images. Now, if you're like most people, you're likely thinking:

I can deploy with the digest - yup, and you can view lots of these: f3c98bff484e830941eed4792b27f446d8aabec8b8fb75b158a9b8324e2da252
I can use semantic versioning, and always bump the number - yup, that sort of works, but read on
I can use my git commit id - that's pretty good as each commit is unique, and you can easily match a container image to its source. However, do you only rebuild your images when your source changes? ...read on

OS & Framework Patching

While I believe in "never say never", using stable tags for deployments are fraught with problems. However, Stable Tags do have their value. Let's consider what happens when developers stop checking in code changes. In the VM world, Ops would continue to patch the running hosts for updates to the framework and patches to the host operating system. And, at some point, those patches may break the apps running on those nodes. One of the many benefits of containers are the ability to isolate the things our apps and services depend upon. Those dependencies are contained within our image, declared within our dockerfile. A multi-stage dockerfile for an aspnetcore app, based linux

FROM microsoft/aspnetcore:2.0 AS base
WORKDIR /app
EXPOSE 80

FROM microsoft/aspnetcore-build:2.0 AS build
WORKDIR /src
COPY HelloWorld.sln ./
COPY HelloWorld/HelloWorld.csproj HelloWorld/
RUN dotnet restore -nowarn:msb3202,nu1503
COPY . .
WORKDIR /src/HelloWorld
RUN dotnet build -c Release -o /app

FROM build AS publish
RUN dotnet publish -c Release -o /app

FROM base AS final
WORKDIR /app
COPY --from=publish /app .
ENTRYPOINT ["dotnet", "HelloWorld.dll"]

In the above example, we can see two images. The aspnetcore-build:2.0 image to compile our code, then we copy the output from our build image to an optimized asnetcore:2.0 image. See: .NET and MultiStage Dockerfiles for more details For the focus of tagging and OS & Framework patching, notice our base image, used for the image we actually deploy. FROM microsoft/aspnetcore:2.0 AS base You can see we're using the stable tag, 2.0. This is where semantic versioning does play a role. The ASP.NET Core team releases new capabilities, indicated by the major version. They shipped 1.0, 2.0 and you can imagine they'll continue to ship new capabilities. In the real world, new capabilities means things change, behaviors change, and thus, you want the development team to make a conscious choice which major version of aspnetcore they depend upon. This basic pattern is used across all major frameworks, java, go, node, etc. While the team focuses on new capabilities, they must also service each version. The aspnetcore image contains aspnetcore, dotnet core, as well as linux or windows server nanao base layers, depending on which architecture you're running. How do you get OS & Framework updates? Should you have to bump your FROM statement for every release? While the team will publish additional tags for minor, non-breaking changes, each team does update their stable major tags. This does mean, depending on when you pull aspnetcore:2.0, you'll get different content. But, that's the intent. Whenever you pull the stable tag, you should get the latest OS & Framework patched image. I say should as not all repo owners do proper maintenance on their images. Why, because it's actually pretty hard. (insert smallest fiddle) . To stay focused on tagging schemes, I'll point you this post on OS & Framework Patching with Docker Containers – a paradigm shift.

What Tagging Scheme Should I Use?

Stable Tags

Stable tags mean a developer, or a build system can continue to pull a specific tag, which continues to get updates. Stable doesn't mean the contents are frozen, rather stable implies the image should be stable, for the intent of that version. To keep it "stable", it's serviced to keep the evil people from corrupting our systems. (ok, evil can also be a simple mistake that has huge impacts) An example: A framework team ships 1.0. They know they'll ship updates, including minor updates. To support stable tags for a given major and minor version, they have two sets of stable tags.

:1 - a stable tag for the major version. 1 will represent the "newest" or "latest" 1.* version.
:1.0 a stable tag for version 1.0, allowing a developer to bind to updates of 1.0, and not be rolled forward to 1.1
:latest which will point to the latest stable tag, no matter what the current major version is.

What's not quite captured above are servicing releases to any of the given tags. As base image updates are available, or any type of servicing release of the framework, the stable tags will be updated to the newest digest that represents the most current stable release of that version. In the above example, updates are provided for the specific versions. In this case, both the major and minor tags are continually being serviced. From a base image scenario, this allows the image owner to provide serviced images. If you really wanted to bind to a specific image, that never changes, you'd set your FROM statement to the digest.

FROM favframework@sha256:45b23dee08asdfls3a7fea6c4cf9c25ccf269ee113ssc19722f87876677c5cb2

However, I'd highly suggest "never" doing binding to a digest. Yeah, I went there - never. If you bind to the digest, how would you get servicing of your base image?

Unique Tagging

Unique tagging simply means, every image pushed has a unique tag. Tags are "never" reused. We've had a number of customers request the ability to enforce this policy, which we'll be adding to ACR. Along with several other policies.

What value should use for your unique tags?

There are a number of options.

~~git commit~~ - works until you start supporting base image updates, where your build system will kick off, with the same git commit as the previous built. However, the base image will have new content. Using a git commit winds up being a semi-stable tag.
date-time-stamp - this is a fairly common approach as you can clearly tell when the image was built. But, how do correlate it back to your build system? Do you have to find the build that was completed at the same time? What time zone are you in? Are all your build systems calibrated to UTC?
digest - it's unique, or should be. But, it's really not usable as it's just toooooo long. And, it doesn't really correlate with anything easily found.
build id - this one is closest to the best as its likely incremental, gives you correlation back to the specific build to find all the artifacts and logs.
- - If your company has several build systems, prefixing the tag with the build system helps you differentiate the API team's Jenkins build system from the Web teams VSTS build system.

Wrapping up

Whew, that was a lot... As with many things, an answer, especially a best practice, depends on ___. Hopefully I've given some context for the various tagging schemes, and the reasoning behind them. To summarize:

Use stable tagging for base images "Never" deploy with stable tags as deploying stable tags leads to instability
Use unique tags for deployments as you likely want deliberate deployments of a consistent version of components. Your hosts won't accidentally pull a newer version, inconsistent with the other nodes when "stuff" happens
Use build id's for your deployed images, so you can correlate it back to the rest of the information in your build system

You may have a good reason for deploying stable tags such as a test environment. But, just realize that images will be updated underneath, and you can very easily spend hours troubleshooting a node that pulled a slightly different version of that tag. Some things to also consider:

Container best practices are evolving
The underlying technology is evolving
You're learning. I like to remind people, learning docker takes a day to realize it's cool, but 3 months to really "get it". Then, you're really just getting started
I'm just a person with an opinion, and I've very possibly missed something. Tell me what you think...

Steve When you stop learning, you start dying... PowerPoint deck with the animations used in the gifs

Relaxing ACR storage limits, with tools to self manage

Steve Lasker — Fri, 02 Feb 2018 20:47:39 GMT

When we created the tiered SKUs for ACR, we built the three tiers with the following scenarios in mind:

Basic - the entry point to get started with ACR. Not intended for production, due to the size, limited webhooks and throughput SLA. Basic registries are encrypted at rest and geo-redundantly stored using Azure Blob Storage, as we believe these are standards that should never be skipped
Standard - the most common registry, where most customers will be fine with the webhooks, storage amount and throughput.
Premium - for the larger companies that have more concurrent throughput requirements, and global deployments.

The goal was never to force someone up a tier, beyond the basic tier. Or worse, cause a build failure if a registry filled up. Well, we all learn. :) It seems customers were quick to enable automation - awesome!. And are quickly filling up their registries. There are two fundamental things we're doing:

ACR will relax the hard constraints on the size of storage. When you exceed the storage associated with your tier, we will charge an overage fee at $0.10/GiB. We did put some new safety limits in place. For instance Premium has a safety of 5tb. If you think you'll really need more, just let us know, and help us understand. We're trying to optimize the experiences, and there are differences we'd need to do for very large registries.
We have pulled up the Auto-Purge feature, allowing customers to manage their dead image pool. Using Auto-purge, customers will be able to declare a policy by which images can be automatically deleted after a certain time. You'll also be able to set the TTL on images. We've heard customers say they need to keep any deployed artifact for __ years. Wow, long time... Eventually, we'll be able to know when it's no longer deployed, and you'll be able to set the TTL for __ units, after the last use.

As customers hit the limits, we wanted to allow customers to store as much as they want, simply charging them for their usage. And enable them to manage their storage, with automated features. When, when, will we get this? The overage meters will start the end of February. Early next week, we'll start the design for the policies on auto-purge. As we know more, I'll post an update. Thanks for the continued feedback, Steve

Working with ACR Geo-replication notifications

Steve Lasker — Mon, 29 Jan 2018 22:09:17 GMT

Azure Container Registry geo-replication enables a single control plane across the global footprint of Azure. You can opt into the regions you'd like your registry to have a network-close/local experience. However, there are some important aspects to consider. ACR Geo-replication is multi-master. You can push to any region, and ACR will replicate the manifests and layers to all other regions you've opted into. However, replication does take some time. Time, based on the size of the image layers. Without an understanding of the latency, it can be confusing and look like a failure has happened. We also recently realized the errors we're reporting can add to the confusion. Lets assume you have a registry that replicates within:

West US
East US
Canada Central
West Europe

You have developers in both West US and West Europe, with production deployments in all regions. ACR uses Azure Traffic manager to find the closest registry to the location of the pull. The registry url remains the same. Which means its important to understand the semantics.

Image flow

From West US, I push a new tag: stevelasdemos.azurecr.io/samples/helloworld:v1. I can immediately pull that tag from West US, as that's the local registry I pushed to. However, If I immediately attempt to pull the same tag from West Europe, the pull will fail as it doesn't yet exist. The pull will fail, even if I issue the docker pull from West US... Hmmm, why? Docker is a client/server model, and the pull is executed from the server. I work in West US. I connected my docker client to the docker host running in West Europe. When I issue a docker pull, my client is just proxying the request to the host, which is running in West Europe. When the pull is executed, traffic manager routes to the closes registry, which is in West Europe. If I remote/ssh into a box in West Europe, or execute a helm deploy from West US to West Europe, the same flow happens. The pull is initiated from West Europe. Which, is actually what we want as we want a local fast, reliable, cheap pull. But, how do I know when I can pull from West Europe?

Image Notifications

ACR supports web hook notifications that fire when images are pushed or deleted. Webhooks are regionalized, so you can create web hooks for each region. In a production deployment, you'll likely want to listen to these web hooks and deploy as the images arrive.

Tag Updates

This is a more interesting conversation, and requires a new post on tagging schemes. Basically, base image owners should re-use tags as they service core versions. When a patch to 2.0 of an image is made available, the 2.0 tag should have the new version. However, a production app should not use stable tags. Every new image to be deployed should use a unique tag. If you have 10 nodes deployed, and one of those nodes fails, the cluster or PaaS service will re-pull the tag it knows should be deployed. (how should you write PaaS service, it's a bit redundant...?) If you re-use tags in production deployments, the new node will pull the most recent version of that tag, while the rest of the nodes are running an older version. If you always use unique tags for deployments, the new node will pull the same, consistent tag that's running on the other nodes.

Tag Replication

ACR geo-replication will replicate tag updates. But, if you're not aware of the flow, you can also get "magical" results. Using the example above, stevelasdemos.azurecr.io/samples/helloworld:v1 has been pushed and replicated to all regions, which are running in all regions. If I push an update to :v1 from West US, and issue a docker run/helm deploy to all regions, my local West US deployment will get the new version of :v1. However, all the other regions will not get the new tag, as they already have that tag running. If you deploy :v2 from West US, and attempt to deploy :v2 in the other regions, you'll get an unknown manifest error.

Geo-replication and Webhook Notifications

When we initial designed the geo-replication feature, we realized this gap and added the regionalized webhooks. We do realize this could be slightly easier as we'd like a deployment to be queued, for when the replicas are deployed. We have some more work to do here, and would encourage feedback on what you'd like to see. What you're scenarios are.

Better Errors

A customer recently raised a ticket with us, as they were receiving a "blob unknown" errors. After digging into the issue, we realized we had an issue with how we cache manifest requests and the ordering of how we sync the blobs, manifests and cache them. A while back, we added manifest caching. Customers were using OSS tools that continually polled the registry, asking for updates to a given tag. They were pinging the registry every second, from hundreds of machines. Each request went into the ACR storage infrastructure to recalculate the manifest. Only to return the same result. We added caching to cache manifest requests, which sped up the requests and improved overall ACR performance, across all customers. While we invalidate the 1hr cache TTL from a docker push, we realized a tag update, replicated across a registry wasn't invalidating the cache. We missed this as we consider it a best practice to never use the same tag, but we also know to never say never. So, we're fixing the cache invalidation when replicated. We will also avoid "blob unknown" errors, when it's really a "manifest unknown" error". In the customer reported problem, we hadn't yet replicated all the content when they did a pull. Because the cache wasn't invalidated, we found the updated manifest, but the blobs hadn't yet replicated. We will make the replication more idem-potent, fix the cache TTL and will more accurately report "manifest unknown" as opposed to "blob unknown".

Preview --> GA

These sort of quirks are why we have a preview phase of a service. We can't possibly foresee all the potential issues without taking really long private bake times, which can miss a market. For customers, they get early access to bits that hopefully help them understand what's coming, so they can focus elsewhere while we improve the service. I just want to end with a thank you for those that help us improve our services, on your behalf. We value every bit of feedback. Steve and the rest of the ACR team.

OS & Framework Patching with Docker Containers - a paradigm shift

Steve Lasker — Wed, 20 Dec 2017 19:29:21 GMT

Thinking of containers, we think of the easy packaging of our app, along with its dependencies. The dockerfile FROM defines the base image used, with the contents copied or restored into the image. We now have a generic package format that can be deployed onto multiple hosts. No longer must the host know about what version of the stack that's used, nor even the version of the operating system. And, we can deploy any configuration we need, unique to our container.

Patching workloads

In the VM world, we patch workloads by patching the host VM. The developer hands the app over to operations, who deploys it on a VM. Along with the list of dependencies that must be applied to the VM. Ops then takes ownership, hoping the patching of the OS & Frameworks don't negatively impact the running workloads. How often do you think ops does a test run of a patch on a VM workload before deploying it? It's not an easy thing to do, or expectation in the VM world. We basically patch, and hope we don't get a call that something no longer works... The workflow involves post deployment management. Projects like Spinnaker and Terraform are making great strides to automating the building process of VM, in an immutable infrastructure model. However, are VMs the equivalent to the transition from vhs to dvds?

Are containers simply a better mousetrap?

In our busy lives, we tend to look for quick fixes. We can allocate 30 seconds of dedicated and open ended thought before we latch onto an idea and want to run with it. We see a pattern, figure it's a better drop-in replacement and boom, we're off to applying this new technique. When recordable dvd players became popular, they were mostly a drop in replacement for vhs tapes. They were a better format, better quality; no need to rewind, but the workflow was generally the same. Did dvds become a drop in replacement for the vhs workflow? Do you remember scheduling a dvd recording, which required setting the clock that was often blinking 12:00am from the last power outage, or was off by an hour as someone forgot how to set it after daylight savings time? At the same time the dvd format was becoming prominent, streaming media became a thing. Although dvds were a better medium to vhs tapes, you could only watch them if you had the physical media. DVRs and streaming media became the primary adopted solution. Not only were they a better quality format, but they solved the usability for integrating with the cable providers schedule. With OnDemand, NetFlix and other video streaming, the entire concept for watching videos changed. I could now watch from my set-top box, my laptop in the bedroom, the hotel, or my phone. The switch to the dvd format is an example of a better mousetrap that didn't go far enough to solve the larger problem. There were better, broader options that solved a broader set of problems, but it was a paradigm shift.

Base image updates

While you could apply a patch to a running container; this falls into the category of: "just because you can, doesn't mean you should". In the container world, we get OS & Framework patches through base image updates. Our base images have stable tags that define a relatively stable version. For instance the microsoft/aspnetcore image has tags for :1.0, 1.1, 2.0. The difference between 1.0, 1.1, 2.0 represent functional and capability changes. Moving between these tags implies there may be a difference in functionality, as well as expanded capabilities. We wouldn't blindly change a tag between these versions and deploy the app into production without some level of validations. Looking at the tags we see the image was last updated some hours or days ago. Even though the 2.0 version was initial shipped months prior. To maintain a stable base image, owners of these base images maintain the latest OS & Framework patches. They continually monitor and update their images for base image updates. The aspnetcore image updates based on the dotnet image. The dotnet image updates based on the Linux and Windows base images. The Windows and Linux base images update their base images, testing them for the dependencies they know of, before releasing. Windows, Linux, Dotnet, Java, Node all provide patches for their published images. The paradigm shift here is providing updated base images with the patches already applied. How can we take advantage of this paradigm shift?

Patching containers in the build pipeline

In the container workflow we continually build and deploy containers. Once a container is deployed, the presumption is it's never touched. The deployment is immutable. To make a change, we rebuild the container. We can then test the container, before it's deployed. The container might be tested individually, or in concert with several other containers. Once there's a level of comfort, the container(s) are deployed. Or, scheduled for deployment using an orchestrator. Rebuilding each time, testing before deployment is a change. But, it's a change that enables new capabilities, such as pre-validating the change will function as expected.

Container workflow primitives

To deliver a holistic solution to OS & Framework patching, there are several primitives. One model would provide a closed-loop solution where every scenario is known. A more popular approach involves providing primitives that can be stitched together. If a component needs to be swapped out, for whatever reason, the user isn't blocked. If we follow the SCC build workflow, we can see the SCC system notifies the build system. When the build system completes, it pushes to a private registry. When the registry receives updates, it can trigger a deployment, or a release management system can sit between, deciding when to release. The primitives here are:

Source that provides notifications
Build system that builds an image
Registry that stores the results

The only thing missing is the source for base images that can provide notifications?

Azure Container Builder

Over the last year, we've been talking with customers, exploring the problem area for how to handle OS & Framework patching. How can we enable a solution that fits within current workflows? Do we really want to enable patching running containers? Or, can we pickup the existing workflows, or the evolving workflows for containerized builds? Just as SCC systems provide notifications, the Azure Container Builder will trigger a build based on a base image update? But, where will the notifications come from? We need a means to know when base images are updated. The missing piece here is the base image cache. Or, more specifically, an index to updated image tags. With a cache of public docker hub images, and any ACR based images your container builder has access to, we have the primitives to trigger an automated build. When code changes are committed, a container build will be triggered. When an update to the base image specified in the dockerfile is made, a container build will be triggered. The additional primitive for base image update notifications is one of the key aspects of the Azure Container Builder. You may choose to use these notifications with your existing CI/CD system. Or, you may choose to use the new Azure Container Builder that can be associated with your Azure Container Registry. All notifications will be available through the Azure Event Grid, providing a common way to communicate asynchronous events across Azure. Giving it a try... We're still finalizing the initial public release of the Azure Container Builder. We were going to wait until when we had this working in public form. The more I read posts from our internal Microsoft teams, our field, Microsoft Regional Directors, and customers looking for a solution, the more I realized it's better to get something out for feedback. We've seen the great work the Google Container builder team has done, and the work AWS Code-build has started. Any cloud provider that offers containerized workflows will need private registries to keep their images network-close. We believe containerized builds should be treated the same way.

What do you think?
Will this fit your needs for OS & Framework patching?
How will your ops teams feel about delegating the OS & Framework patching to the development teams workflow?

Steve