2 - The Beginning

Trey Research is being forced to reexamine how it creates and releases software. As with many organizations, they're thinking about change because the way they currently do things is damaging their business. They want to release reliable products quickly in response to market demand. This sounds reasonable but right now they don't have the processes and tools in place that will allow them to accomplish this goal.

This chapter focuses on ways to analyze how your business creates and delivers software. Some companies may never have clearly formulated how an idea is transformed into a product or service that's released to customers.

Of course, it's entirely possible to move from confusion to clarity. One way to begin sorting out a development process requires nothing more than a wall, sticky notes, and input from everyone involved. This is how the Trey Research team starts.

At this point, Jin is starting to wonder if he's made a big mistake.

Jin says:

Thursday, July 11, 2013

Complete chaos? Lost at sea? Those are nice ways to describe what's going on here. There's enough work to keep us all busy for months but no one has a good reason for doing most of it. What's worse is that no one even knows why some of the work is even there. On top of that, no one agrees with anyone else about what's important. Everyone has their own agenda. Maybe I should quit, or consider another, easier job, like herding cats, or maybe I should just look at it all as a challenge. No pain, no gain, right? I think I'm going to take all those sticky notes and put them into TFS so we have an actual backlog.

Here's the result of Jin's efforts. This is the backlog for the Trey Research application. It shows all the tasks that still need to be done.

After the dust settles and they take a look at the backlog, the team discovers that there really is some order under all the seeming chaos. They find that there are eight stages to the Trey Research development process. Here are the major stages the Trey Research team identified.

1. Assess. Someone identifies something that could cause a change to the software. For example, an end user might make a suggestion, the marketing department might make a request after they've done some research, or the idea could come from anyone on the team. Some reasons to make the change might be to take advantage of a new opportunity, fill a business need, or fix a bug. Basically, anything that affects the code needs to be assessed to see if it's a reasonable request.
2. Approve. A product owner makes revisions and approves the change.
3. Specify. Someone analyzes the change and writes a high-level specification. There might be an associated user story or a bug report.
4. Plan. The team that's going to implement the change decides on its priority, estimates how long it will take to write it, and adds it to their schedule.
5. Build. The team writes the code, builds it, and does the basic build, unit, and commit tests. Once any bugs are fixed they can move on to the next step.
6. Deploy. The team deploys the new version of the software to a staging environment.
7. Test. The team performs more functional tests and fixes the bugs.
8. Release. The team releases the software to the production environment. If any problems occur here, they need to be fixed.

The team summarizes this information by drawing a value stream map. The map represents the flow of requests through the business and relates it to efficiency and wait time. Efficiency is the amount of time when value is being added. Wait time is a delay that adds no value. Here's the value stream map for Trey Research as it exists today.

As you can see, it takes about nine weeks for a single feature to move from being assessed to being released. The metric for how long it takes for the entire process is the lead time. More than half of that time is spent moving from build to release. The build phase alone takes two weeks. So, not only is the process flawed because it allows unreliable software to be released, but it's slow. It takes more than a month to discover all the errors.

Next, the team decides to look at each of the eight phases they identified and to list all the steps that comprise each one. To do this, they get a lot more sticky notes, go back to the wall, mark off eight columns, and then add a few more columns just in case they missed something. Here's the final version.

The number in each column represents the work-in-progress (WIP) limit. It's the most items the column can hold. The WIP number will be adjusted later as a result of what they learn as they actually do some work. They've also added some buffer columns to designate the flow of work between the stages, still with respect to the WIP limit. When the work is likely to be blocked by external reasons they use one of the Ready for… buffer columns. They use a Done buffer column if they can remove the blocking factor by themselves.

Some people call such a display a kanban board. The word kanban is Japanese and means signboard or billboard. Toyota began using kanban in the 1940s. It developed the idea by watching how supermarkets stocked their shelves. Toyota applied the same principles to its factories. Beginning with an agreed upon number of physical cards, it attached one card to each new piece of work. The card stayed with the work as it progressed through the factory. New work could start only if a card was available. If there were no more cards, the work waited in a queue. When one piece of work was finished, its card was attached to a piece of work in that queue. Kanban is a pull system because work is brought into the system only when there's capacity to handle it. If the predetermined number of cards is set correctly, then the system can't be overloaded.

Kanban and kanban boards can be easily adapted to other industries, including software development. For example, practitioners of lean software development and Agile often use kanban boards to make the organization and status of their projects visible to all team members. A kanban board is useful no matter the methodology the team actually follows. For example, the kanban board we've just shown you could be adapted to work for a team that uses Scrum. The team could certainly keep the Specification, Planning, Development, Deployment, and Test columns. Even the Release column could be included if they want to introduce some DevOps ideas into their work. The column could represent developers and operations people working together to release the software. A Scrum team might change the columns to better reflect the nonsequential Scrum approach, or even have a different board for the sprint backlog. Still, the kanban board would be useful for visualizing the entire project, end to end.

Here's Jin's reactions to what's been happening.

Jin says:	Friday, July 12, 2013
	The value steam map helped, and the Kanban board looks good, too. At least we can all see what's going on, and that's a good first step. I'm going to do the same thing for the Kanban board as I did for the sticky notes. I'll put them into TFS. I'm hoping I can convince everyone else to help me keep the board updated. Maybe when they start to see some progress, they'll be more willing to join in.

Here's the Kanban board that Jin created in TFS. The fractional numbers in the column headings (3/3, for example), show each column's WIP limit. For each backlog item in the first column, you can see all the tasks (the blue boxes) that comprise the item. Each task's status is shown by the column it's in.

Jin however, is nervous.

Jin says:	Friday, July 12, 2013
	On Monday, we start the next iteration, and it's a three week sprint. That's huge, by my standards, but it's the best I could get. Their first release took two months! I hope the work we've done so far will help us to stay focused and meet our goals.

When the rest of the Trey Research team members look at their Kanban board, they're more hopeful. They realize that they can make real improvements to the last four stages of the value stream. These stages comprise the release pipeline.

What is a Release Pipeline?

A release pipeline is an implementation of the process a business follows to create and deliver software. The pipeline begins at the point when any change (for example, a new feature or a fix) is introduced into the code base, and ends when the change is delivered to users. As the change moves through the pipeline, it undergoes a series of activities, each of which helps insure that the final software is production ready and conforms to the specified requirements. A release pipeline is automated to some degree. Automation means that (some) activities that occur in the pipeline and (some) movement from one end of the pipeline to the other happen without human intervention. A common example of automation is continuous integration, where builds are automatically triggered by check-ins and some set of tests run after the build is complete. While you may not be able (or want) to automate the entire pipeline, automating as much as is feasible can greatly improve how you deliver software.

What Are Release Pipeline Stages?

Stages are the building blocks that form a pipeline. Each stage contains a set of related steps (or activities) that occur during the software development process. These stages are in place to ensure that any change introduced into the pipeline will be good enough to be released into production. Commonly used pipeline stages are the commit stage, the automated acceptance test stage, and the release stage. Any pipeline stage can either pass or fail. If it fails, the pipeline should completely stop until the problem is resolved.

What Are Release Pipeline Steps?

A step is an action that's contained within a stage. Stages are typically comprised of several steps. The build stage, for example, encapsulates steps such as creating a build, running tests, and performing code analysis. Steps can be either automated or manual. There are many different steps that can occur in each pipeline stage. A few examples are building the software, running unit tests, provisioning an environment, running automated acceptance tests, and manually performing exploratory testing.

What Are Release Pipeline Instances, Changes, and Versions?

An instance of a pipeline is an actual occurrence of that pipeline. When a pipeline is triggered by an event (typically, when there is a check-in to the version control system) a pipeline instance is created. Each instance should be uniquely identified. The best way to do this is to name the pipeline after the check-in that triggered it. You could, for example, use the version control changeset or the version number of the code. You can use this identifier to track the stages that run inside the pipeline instance and to discover their outcomes.

Changes and versions are used interchangeably in the context of continuous delivery, and in this guidance we use both. A change or a version is some modification to the code or supporting artifacts that is checked in to version control. From the perspective of continuous delivery, every version is a candidate for release into production. Any instance of the pipeline validates only one version of the software.

What Tools Does a Release Pipeline Require?

You will use a variety of tools to create and run a release pipeline. Here's a high-level overview of the sorts of tools you'll need.

• Tools that support the pipeline. These tools run the pipeline, define and trigger stages, orchestrate the overall process and gather information for metrics. Specific examples of this type of tool are Team Foundation Server (TFS), Jenkins, and Go.
• Tools that support pipeline steps. These tools set up and run specific steps inside the pipeline stages. They can include scripting and automation languages and technologies, tools to set up and run environments, and tools to build and package software. Specific examples include PowerShell, MSBuild, MSDeploy, NAnt, SystemCenterVirtualMachineManager****(SCVMM), and Lab Management.
• Other supporting tools. These tools include, for example, tools for software and script development, tools for testing and debugging, integrated development environments, tools for database management, and tools for version control. Specific examples include Microsoft Visual Studio, Microsoft Test Manager, Microsoft SQL Server Management Studio, TFS, and Visual Studio Profiler.

What Stages Do I Need in a Release Pipeline?

How you decide what stages your own release pipeline needs is determined by your business requirements. For example, if your business runs a website with millions of simultaneous users, you would certainly want to include a stage that tests for capacity. In many cases, the minimum set of stages required to ensure that the software is production ready, at least from a functional perspective, would be a commit stage, an acceptance test stage, a UAT stage, and a release stage.

Analyzing the Trey Research Release Pipeline

After they formalized the value stream map, the Trey Research team decides to take a close look at its last four stages–the build, deploy, test, and release stages that actually produce the software. These four stages are what comprise the Trey Research release pipeline. In the end, the team wants a diagram that completely defines it. There are many questions to be answered before they understand how their pipeline works. Here are some of them.

• What's in the version control system?
• How many code branches are there?
• What happens in each stage of the pipeline?
• How does code move from one stage to another?
• How many builds are there?
• How are the builds defined?
• Where are code artifacts stored?
• How many environments are there?
• How are environments provisioned?

After a lot of work they came up with a diagram. Here's what they currently have.

Let's walk through their pipeline to see how it works.

The Version Control System

Trey Research uses TFS as its versioning system. It contains all the application code but no other artifacts, such as configuration files or scripts.

The main branch of the versioning system holds the code that's considered stable enough for the next version of the application. Iselda, the tester, performs functional tests on this code to see if it performs as it should. All new features are developed in the dev branch.

In terms of releasing code, they know that it's important to always be able to identify the code that produced the binaries currently in production. They want to ensure that any bugs found in the production code can be fixed without introducing untested features or unknown problems. To be certain that they always have the code that generated the currently running binaries, they store it in the release branch. They only need one release branch because there's only a single version of the application in production at any time. They update this branch after each release by merging the code stored in the main branch and overwriting what's in the release branch.

The Release Pipeline

The Trey Research release pipeline has four stages: build, deploy, test, and release. The stages are sequential so every stage (except the first) depends on the previous stage to complete before it can do its job. After one stage finishes successfully, any team member can manually start the next stage. Stages themselves are composed of various steps, such as "perform continuous integration," and "perform code analysis." The team uses TFS in the build, deploy, and release stages.

The Build Stage

In this book, we make a distinction between creating a build and running tests. (Some people combine the notion of building and the notion of testing under the single term "build.") Creating a build means to prepare all the artifacts that will be run by the steps included in a stage. In other words, a build does more than compile the source code. It can also include activities such as copying files, references and dependencies, (these include supporting files such as images and software packages), and signing assemblies. Unit tests are run after the build is available.

At Trey Research, the build stage uses the code that’s stored in the development branch. The build stage is defined by a TFS build definition that's based on the default template. The build definition includes information such as what should be built, when a build should run (that is, what triggers a build) the workspace, and the build controller you want to use. Here's an illustration of how a build occurs.

When it's determined that a build should occur (this is a part of the build definition) a team project sends a request to the build controller's message queue. (The build controller accepts build requests from any team project in a specified team project collection.) The controller retrieves the build definition and executes it.

Along with processing workflows, controllers perform some other work such as determining the name of the build and reporting the build status. They also select and manage the services of the build agents.

The build agent actually performs the build. It does all the processor-intensive work. It calls MSBuild to compile the code and run the tests. After the build agent successfully completes its build, it archives the build results to a file share called the drop location.

Trey Research's Build Definition

Trey Research uses Team Foundation build definitions that are based on the default template. Build definitions are created from within Visual Studio. If you want to recreate the Trey Research build definition, see exercise 4 of Lab 1: Creating the Trey Research Environment.

Before a build happens, Paulus, the developer, brings up the application on his development machine and performs some basic tests. He does this because it's easier to run the application in Windows Azure and Windows Phone 8 emulators than to deploy it to the real platform. Once he feels confident about the code, he checks it in to the dev branch, which triggers the build stage. Team members have set a TFS alert and they receive emails when a build stage ends.

The Deploy Stage

After all the steps in the build stage are complete, one of the team members uses Source Control Explorer to merge code from the corresponding development branch to the main branch. The transition from the build to the deploy stage is manually triggered. It only runs when someone explicitly requests it, perhaps through Visual Studio or the command line.

The deploy stage's job is to send the build artifacts to the test stage. However, it doesn't use the artifacts that were generated by the build stage. Instead, it retrieves the merged code from the main branch of the versioning system. It then performs another build. This stage also uses a Team Foundation build definition that's based on the default template.

The Test Stage

The test stage has its own environment. It uses the binaries and artifacts created in the deploy stage. Right now, there are only some manual acceptance tests, which aren't closely tied to user stories or requirements. There's also no dedicated tool for managing and executing test plans. Instead, Iselda, the tester, uses Microsoft Excel spreadsheets to keep track of what's going on. Moving from the deploy stage to the test stage requires a manual trigger. Iselda connects to the test environment and then runs all her tests manually.

One known issue with the test stage is that there's no UAT environment. Iselda wants one but Trey Research is on a tight budget and environments are expensive to set up and maintain.

The Release Stage

After the test stage is complete and the bugs are fixed, one of the team members uses Source Control Explorer to merge code from the main branch to the release branch. The transition from the test stage to the release stage is manually triggered. The release stage builds the code that was merged from the main branch to the release branch. Again, there are no tests to confirm that the build was successful. After the code is built, it's manually deployed to the production environment where manual smoke tests are performed as well as user acceptance tests (UAT). The release stage also uses a Team Foundation Build definition based on the default template.

The Environments

Trey Research currently has three environments: development, test, and production. Each of them is configured manually. There is no automated configuration. Each environment has a different URL that the Windows Phone 8 app uses to connect to the appropriate Windows Azure service.

The Problems

Finally, after analyzing the pipeline, the team begins to list the problems they're having. Right now, there are so many that it's overwhelming. Here are just some of them.

• It's not clear which binary is deployed to each environment, or what source code generated it.
• There's nothing that defines the quality of the release. No one knows if it meets customer expectations or includes the agreed upon functionality.
• There are versioning inconsistencies. Incompatible versions are deployed across different environments and devices.
• There's a lot of time wasted, confusion, and errors introduced by manually moving from one stage to another.
• There are many unexpected bugs because the pipeline builds and deploys different binaries for the same version or change. These bugs make it difficult to release quickly.
• The deployment process is slow and error prone.
• They don't always work on the correct version of the application so they find and work on bugs they've already fixed.
• Regression testing is slow and error prone so they don't do it as often as they should.

Everyone is back to feeling overwhelmed.

Jin's discouraged, not just because of the chaotic state of the pipeline, but because of everyone's attitude.

Jin says:

Monday, July 15, 2013

After a lot of arguing, we did set up a backlog but what a struggle! Iselda wants to test everything after the iteration finishes. She says she has lots of other work to do but Zachary made it clear that the Trey Research app has top priority. She finally came around and is going to test along with development and not wait until they finish. Raymond is really difficult. He didn't understand why he had to come to our meetings when there are already plenty of problems in the server room. He left as soon as he could. We did manage to add a couple deployment tasks for him to the backlog. I told everyone I'd take this iteration's backlog and all the tasks and put them into TFS. Paulus said (and I know everyone else was thinking the same thing) that it's a waste of time and that we should be coding. They'll see.

Here's the result of Jin's work. The user stories are in the first column. The blue boxes are the tasks that make up each user story.

Still, even with all the preparation, there are many problems. By the middle of the iteration, Jin sees trouble.

Jin says:

Wednesday, July 24, 2013

It's clear we won’t reach our sprint goal by August 2nd. That's what the iteration board and the burndown chart tell us. One small victory is that Paulus admitted that it's useful to keep information in TFS but none of them help keep the board up to date. At least we know that whatever's going to be delivered is actually finished, or "done-done" as some folks say. Of course, that's from the developer's side. I still don’t know how we're going to coordinate with Raymond to get our app into the production environment and out to our users.

Here's the latest iteration board. It shows the user stories in the first column. The blue boxes show the tasks that comprise each story. As you can see, there are only a few tasks in the Closed column. All the others are either in the New column, which means that they haven't been started or in the Active column, which means that they're not finished. Because the team is already in the middle of the iteration, it seems clear that they won't meet their goal.

Jin sees even more bad news.

Jin says:	Wednesday, July 24, 2013
	We've ignored the WIP limits in our buffer column for development. Nobody was concerned or saw it as a problem. They didn't seem to mind at all. Maybe they think it shows how hard they can all work. Right now, everyone's focused on producing a release, but we really need to change the way we do things around here.

Here's what the product backlog shows. There are four items in the Develop—Ready For Develop column when there should only be two.

At the end of the iteration, the news still isn't good.

Jin says:

Friday, August 2, 2013

We completely missed our forecast. We did manage to deliver a working version of the basic services, along with a simple desktop client. But we're missing many features we planned for this milestone, and I don't think anyone will buy what we have now. There were even some features that we developed that weren't delivered to customers. They stayed in the Release column of our board, waiting for the binaries to be promoted to production. I won’t blame Raymond, because this is the whole team’s fault. But I do say he's a part of the team, no matter how upset he gets when I mention it. If we don't all work together, we'll never get rid of the bottleneck just before the release column, where items keep piling up.

Here's what the Kanban board looks like at the end of the iteration.

Here are Jin's final comments for this iteration.

Jin says:

Friday, August 2, 2013

Zachary's really angry. It took almost five weeks to get that little green square at the end of the board, and it's all alone. It doesn’t matter if there are four more items almost there. If they're not deployed to production, nobody cares. Stakeholders won’t support this much longer. The one thing Zachary did like was that he could see the status of every work item. He said that now he really knew what was going on. He liked it so much that he told the rest of the team to help me keep it up to date. Everyone looked at their shoes.

Summary

This chapter talks about ways to understand your development process, from the time you have only an idea for a feature or product, until the time your idea is realized and you can deliver it to your customers. Tools aren't what matter at this stage. What matters most is that everyone involved have a voice and participate in the analysis. The procedure we use has five phases.

1. Start with the stated goal of understanding how you develop your software. You only need some sticky notes, a wall, and your team members. Let everyone participate, adding steps that they think are important. Try to identify the most general activities.
2. Once you've got a sense of the major activities that happen (for example, in this chapter, Trey Research narrows it down to eight), formalize your system with a value stream map. Using the map, relate the activities to how much time you spend accomplishing them and to how much time you're blocked.
3. You might want to create a Kanban board to get a more granular understanding of what happens during each of the major stages you've identified.
4. Focus on the release pipeline itself. Draw a diagram that shows how the bits flow from one stage to another. List the steps in each stage, the tools you use, the environments each stage depends on, and how you store and keep track of builds and all other artifacts that affect your development process.
5. List the problems you're having.
6. (Optional) Take a deep breath.

We also talked about release pipelines. A release pipeline is an implementation of the process a business follows to create and deliver software. It consists of stages. Stages are the building blocks that form a pipeline. Each stage contains a set of related steps that occur during the software development process. Steps are actions that are contained in stages. A new version triggers an instance of a pipeline. A version is some modification to the code or supporting artifacts that is checked in to version control.

For the Trey Research team, life is still very difficult. They've made progress in terms of using TFS to help them organize their work and to gain visibility into their release process. However, they're still missing major goals and not shipping what they should.

What's Next

In the next chapter, the Trey Research team, although initially overwhelmed, starts to put the pieces together. First, they figure out how to set some priorities. They also start to think about orchestration. By orchestration we mean the arrangement, coordination, and management of the release pipeline. The team makes a decision and decides they're going to gradually move toward a continuous delivery pipeline. They'll build it using the TFS default build template and the Lab Management default template, along with some supporting tools.

More Information

There are a number of resources listed in text throughout the book. These resources will provide additional background, bring you up to speed on various technologies, and so forth. For your convenience, there is a bibliography online that contains all the links so that these resources are just a click away. You can find the bibliography at: https://msdn.microsoft.com/library/dn449954.aspx.

The ALM Rangers have put together a collection of guidance and labs to help you use TFS for application lifecycle management. For more information, go to Visual Studio Team Foundation Server Requirements Engineering Guidance at http://vsartfsreguide.codeplex.com/.

The ALM Rangers have put together a collection of guidance and labs to help you understand how to customize and deploy Team Foundation builds. For more information, go to Team Foundation Build Customization Guides at http://vsarbuildguide.codeplex.com/.

To understand how to use the TFS Kanban board to understand the status of your project, see Work on the Kanban board at https://tfs.visualstudio.com/learn/work-on-your-kanban-board.

To learn how to customize the TFS Kanban board, see Kanban customizable columns, "under the hood" at https://blogs.msdn.com/b/visualstudioalm/archive/2013/03/04/kanban-customizable-columns-under-the-hood.aspx.

The hands-on labs that accompany this guidance are available on the Microsoft Download Center at https://go.microsoft.com/fwlink/p/?LinkID=317536.

Next Topic | Previous Topic | Home | Community