Release: Final Testing and Support
Applies to: Agile
Authors: Greg Smith and Ahmed Sidky
.jpg "Referenced Screen")
Get this book in Print, PDF, ePub and Kindle at manning.com. Use code “MSDN37a” to save 37%.
This topic contains the following sections.
- Communication and Training
- Ready to Release
- Enough Planning; Let’s Deploy
- Key Points
- Copyright
Communication and Training
Agile development understands the importance of communication, and that is why you put team members in close proximity to each other. It’s also important to communicate with individuals who aren’t a direct part of the project team. Let’s look at a few typical messages communicated during a project.
Table 21.2 lists potential audiences and messages for a given project. To give you context, we’ve listed the audiences and messages that Acme Media addresses during the Auctionator project.
Acme Media has a diverse group to communicate with, but you may have additional groups unique to your project. For example, if you work in a regulated industry, you may have to communicate to governing bodies such as the FDA and OSHA. You may also need to create communications that validate Sarbanes-Oxley (SOX) compliance for your project.
Table 21.2 Acme Media outlines its training and communication plan before going live.
Audience |
Message or training needed |
End users:
|
|
Paying customers:
|
|
Internal employees and stakeholders:
|
|
Support groups:
|
|
Stakeholders’ sponsor:
|
|
The Auctionator is a medium-size project for Acme Media, so the communication required is also at a medium level. Acme chooses the level of documentation needed for each project based on variables such as audience size, project duration, and the level of ceremony required. You should do the same on your projects.
Ready to Release
Now that you have all the support areas and processes ready to go, you need to make the final decision about going live and then plan the deployment steps in detail.
Deciding to Go Live
Although every iteration you deliver is deployable, teams often need to obtain approval from customers, stakeholders, support groups, network teams, testers, and change-management groups before deploying code to a production environment.
We’ve covered common go-live criteria in this chapter. You’ll consider code stability, outstanding defects, readiness of support groups, system performance, and user readiness. You can analyze these areas from a statistical perspective, but the ultimate decision may be based more on how you feel about the project than on empirical statistics.
Acme Media analyzes the state of its project. The team, the customer, and the sponsor all agree to go live, noting only one concern: the auction functionality doesn’t work correctly with an older version of Firefox. The team reviews server-log information and determines that few users have the old Firefox version. They also note that the version isn’t included in their list of officially supported browsers. You can see Acme Media’s implementation checklist in Table 21.3.
This discussion may seem simple for a go-live discussion, but it’s important to remember that Acme Media is working in an Agile environment now. The company has been building and stabilizing the product with frequent builds and iterative creation of features. Acme works to make each iteration solid, so a go-live conversation is closer to a formality than a requirement. There shouldn’t be many surprises or unknowns at this time.
Now that you know you’re going forward, let’s plan the deployment steps.
Table 21.3 An implementation checklist helps everyone grasp the state of the project and contribute to the go-live decision.
Questions |
Yes |
No |
Have all the requirements in scope been met? |
x |
|
If not, has agreement been reached for those elements not satisfied? |
||
Does the product support Segregation of Duties -- the division of roles and responsibilities to reasonably prevent a single individual from subverting a critical process? |
x |
|
Are there any business or regulatory changes that would prevent implementation of the product(s) of this project? |
x |
|
Do you agree to accept all documented risks associated with deploying this product? This may include potential risks to existing systems or business operations. |
x |
|
Do you understand the outstanding defects and their possible consequences as described in the Technical Test Results (as applicable) and in the User Acceptance Test Results? |
x |
|
Are all employees who will use, support or are otherwise affected by the product(s) prepared for implementation? |
x |
|
Do you agree to move the product(s) into Implementation? |
x |
|
Has all User Acceptance Testing been completed? |
x |
|
Have all business functions specific to this application been tested? |
x |
|
Have all SOX control tests relevant to this application been tested? |
x |
|
Have all non-functional tests been completed? |
x |
|
Has Performance Testing been completed, including load, stress and capacity testing? |
x |
|
Has Security Testing been completed? |
x |
|
Has Recoverability Testing been completed? |
x |
|
Has Usability Testing been completed? |
x |
|
Have workarounds for known defects been documented and communicated to all affected parties? |
x |
|
Have all the required Business Continuity (BC)/ Disaster Recovery (DR) deliverables been successfully completed? |
x |
Planning the Deployment Steps
You may remember that during the development iterations we discussed code completeness. You make sure the code can be demonstrated, that it is tested, and that it supports nonfunctional requirements. Many features also require you to design a process for their deployment. This process may require that you write scripts and configure the system to support the new feature. In our experience, deployment scripts and processes are a part of the deliverables during a development iteration.
When you deploy your iterations to the production environment, you need to create an overall process and sequence for deploying the features.
Note
Yes, You Can Deploy Instantly
You may be reading this chapter and thinking, “This is ridiculous. We release code every few days!” We’re sure this is true for many people.
This chapter is focused on a medium-size project, and we’re implying a certain level of ceremony to give you a feel for the options available during a deployment. We’ve worked in environments where we deployed new features daily (an online newspaper), and we’ve worked on projects where it took a year to deploy (software for medical devices).
Deploying quickly is a good thing. The sooner you get your code in place, the less chance there is that you’ll miss the need or opportunity. But if you work on larger projects, you may find interdependencies between features, and deploying may not make sense until the bulk of these features are in place. We’re demonstrating this model with Acme Media in hopes of helping you when deployment becomes more complex due to a larger audience or dependencies on outside groups.
Many times your architecture will influence the process you use. This is true with Acme Media. Acme developed its deployment process around two realities of the production environment. Let’s look at the realities Acme Media had to take into consideration.
Deployment Considerations
Acme Media developed its deployment plan around the realities of the company’s environment. Let’s look at unique areas for Acme Media.
Architecture
Acme’s architecture has two elements that influence the deployment plan:
Disaster recovery. Acme maintains a DR environment to fail over to in case of an emergency. This backup system is maintained in a data center across town. Acme always deploys to the backup center first as an additional test of the code and as a test of the deployment plan.
Load balancer. Acme uses a load balancer to equalize the load across the company’s servers and to ensure the site stays up if one server goes down. Acme uses the server pool during deployment to allow the site to stay up while new code is installed onto each server.
Safe Window to Deploy
Acme must also consider other variables when choosing the day and time for deployment. The team goes over the following checklist to determine the best window:
Holidays. Acme avoids deploying on holidays because support groups aren’t available.
Other deployments. The team chats with other departments to make sure they aren’t doing deployments or infrastructure work that could block a deployment.
Resource availability. Acme has to make sure the types of employees needed for deployment aren’t on vacation or in training during the deployment window.
Blackout windows. Acme has windows of time where deployments aren’t allowed. These windows usually correspond to month-end or year-end financial processing.
User activity. Acme times releases so there is minimal affect on users. This means avoiding peak usage windows in case an issue is encountered.
The Acme Media team doesn’t have to worry about migrating or converting data when they deploy, but they do have to create a plan for phasing out and decommissioning the old merchandise site.
Migration and Conversion
Although it doesn’t affect Acme Media, we’ve worked with teams that deploy only when the customer is available to test the product in the production environment.
Creating a Deployment and Backout Plan
Acme Media creates deployment scripts and processes as a part of code delivery during the development iterations. The team reviews these scripts during deployment planning and also lays out a logical sequence for running the scripts.
Acme uses a collaborative process to create a deployment plan for each release. Wendy, the project manager, organizes a meeting to discuss the timing for deploying the code.
In the spirit of being Agile and productive, Acme doesn’t follow the same exact process for each deployment. For example, the load-balancer architecture allows Acme to deploy to the production environment without blocking user access to the system. Individual servers can be removed from the load-balancer pool and then reinserted after they’ve received the new code. When the team has a slight change to make to the production environment, they do it during normal hours and without splashing the site (see Table 21.4).
When the Acme team outlines the deployment plan for the Auctionator, they consider the complexity of the deployment scripts, the time needed to put the code into production, and the risk associated with the deployment. They also consider the complexity of backing out the code if an issue is encountered. They decide to do a deployment in the evening when website traffic is down, so that if an issue is encountered there will be minimal impact to site visitors.
Table 21.4 Acme Media’s deployment plan for the Auctionator.
Time |
Activity |
Expected result |
Action |
Go/no go decision |
Owner |
5:00PM-7:45PM |
Backup SQL Server databases |
Databases backed up successfully |
DBA will verify that database backup is completed. |
Stop deployment |
Aaron |
1:00PM-4:00PM |
Begin Server pre-validation (restart) of all servers |
All servers restart successfully |
Ensure servers are ping-able/ apps online |
QA confirms the functionality works via BVT |
Jim/Gina |
7:45pm |
Run SQL Script and Remove SQL Hardening |
Script runs successfully |
DBA verifies script completes without errors |
Re-run script |
Aaron |
8:00pm |
Remove ‘Set A’ Production servers from application pool |
Set A servers are not accessible via FQDN |
Ensure all Set A servers are not in the load balancer pool |
Confirm all Set A servers are not accessible |
Jim/Gina |
8:05pm |
Clean-up and Deploy OLA code to ‘Set A’ Production Servers |
New OLA Code functioning |
Ensure correct OLA version was installed to each server |
QA confirms new functionality works as expected |
Jim/Gina |
8:50PM |
Add Set A Production Servers back into application pool |
New OLA Code functioning |
Ensure all Set A servers are back in the load balancer pool |
QA confirms new OLA functionality works as expected |
Jim/Gina |
9:30PM |
Remove ‘Set B’ Production servers from application pool |
Set B servers are not accessible via FQDN |
Ensure all Set B servers are not in the load balancer pool |
Confirm all Set B servers are not accessible |
Jim |
9:30pm |
Clean-up and Deploy OLA code to ‘Set B’ Production Servers |
New OLA Code functioning |
Ensure correct OLA version was installed to each server |
QA confirms new OLA functionality works as expected |
Jim/Gina |
9:35 PM |
Apply redirects to old Merchandise site URLs |
|
|
Verify redirects to Auction site. |
Ryan |
10:35 PM |
Verify merchant ads are serving correctly |
Gina |
|||
12:10AM |
Reapply SQL Hardening |
|
|
|
Gina/ Aaron |
12:30AM |
All sets added back in. Announce deployment. |
Wendy |
Acme Media will also have a chance to test the deployment plan when the team deploys the Auctionator to the DR environment.
Reducing Risk with a Pilot
Acme Media minimizes deployment risk by deploying to the DR environment first. Many teams don’t have a DR environment and need another way to test with minimal risk. For these teams, a pilot deployment or soft launch can start an iterative deployment process.
You can perform a pilot by deploying to your production environment but limiting access to a small audience. The small audience provides feedback and is aware of the risks that come from piloting, such as limited support and the possibility that their data could be lost. A pilot is also frequently used on projects where quality is the number-one priority.
Enough Planning; Let’s Deploy
As we mentioned in an earlier sidebar, deployment can happen relatively quickly on smaller projects or where the architecture lends itself to a quick process. The Auctionator will take several hours to deploy, so the team will do it in a collaborative fashion.
Acme Media’s team members sit relatively close together at their desks, but for deployments they all go into a conference room and sit at a Table face to face. This setup allows the team to tell each other when they’re finished with a step during the deployment. It also accelerates troubleshooting if an issue is encountered. The whole team can hear the issue at once, and each team member can investigate the issue from a different angle.
If you saw the movie Apollo 13, you may remember the scene in which the crew experienced an issue and alerted ground control. In real life, flight director Gene Kranz asked each group to analyze the problem from their perspective. One person investigated a potential sensor malfunction, another person looked to see if the issue was consistent across all systems, and another person tried to Figure out what was going on by studying the biosensors on the astronauts.
When we participate in deployments, we see similar troubleshooting methods when we encounter an issue. For example, if a web page isn’t displaying correctly, the user experience designer verifies that the stylesheet installed correctly, a network engineer verifies that the web servers are running, and an implementation engineer verifies that the database scripts run correctly.
When Acme Media completes its deployment, Wendy sends out a notice to the company so everyone can prepare to support the new application.
Celebrate!
Acme Media doesn’t have a history of celebrating deployments. Many of the company’s projects never reached completion, or teams weren’t proud of the work they did release.
The Auctionator is one of Acme’s best accomplishments. For once, Acme isn’t deploying features that provide minimal value, and they’ve also hit their timeline for deploying the critical features. Acme is also happy because the whole team agrees that the project was valuable and worth pursuing.
When you deliver a project like the Auctionator, you need to celebrate. Although a lot of people say “it’s just work,” we all put a lot of personal effort into delivering a project, and we take pride in what we’ve overcome and what we’ve delivered. It’s important to celebrate the achievements of the team, the issues that were resolved, and the fact that you delivered the functionality that was needed at the end. It’s also a moment for management to tell employees that their work is valued and important.
Every company has a different way of celebrating, whether it’s a party or an offsite gathering. The critical piece of the celebration is timing: you should celebrate while the project is still on everyone’s mind, before you shift to the next project. We’ve worked on several projects where we couldn’t find time to celebrate at the end or delayed the celebration for one reason or another. In those instances, the celebration seemed a little hollow and staged, and we’d forgotten some of the hard work we performed and why we were entitled to a celebration.
You’ve probably been to celebrations where a group of people congregated in a corner and started whispering, “Why are we celebrating? Were we really successful?” Many projects take off without clearly defining what success means.
Acme Media defined success as stopping the steady decline of customers to eBay and Craigslist and increasing site revenue by adding merchant target advertising. Acme celebrates a few days after deployment and doesn’t have significant data to determine if customers are returning. But the company does have a list of merchants who’ve signed contracts for targeted advertising. These merchants were pursued as the project was being completed, and the initial number that signed up for the advertising program is in synch with the numbers Jay envisioned for the first month of the new site.
Key Points
The key points from this chapter are as follows:
You create software in deployable units during each development iteration, but you don’t deploy after every iteration because of the ceremony required to release. Your environment may make it easy to deploy code, but many software projects require training, communications, and final acceptance testing before features can be released.
Code can be released to support a deadline, according to a predetermined release schedule, or when the team and customer agree there is enough value to deploy.
For many companies, additional testing and validation must take place before releasing. This can include final functional testing and testing of nonfunctional requirements such as system response time.
Many companies overlook validation of nonfunctional requirements. Such companies frequently fail dramatically during a product release.
Many projects require the creation of new processes and documentation for support. You create the support processes and train your support team on them before deployment.
Many teams need to document their go-live plan, especially when deploying new technology. This plan integrates the deployment and configuration steps created during each iteration.
You may find it valuable to iteratively roll out your project to the audience it’s intended for. A pilot or beta group can assist with this iterative process and reduce deployment risk.
If your team doesn’t use a team room on a daily basis, establish one for the day of deployment. Having everyone co-located during deployment helps resolve issues quickly by collaboratively working the issues.
Always celebrate your projects quickly after deployment. It’s easy to get distracted once you go live, either due to go-live issues or pursuit of the next project. If you don’t celebrate, you’ll dilute your accomplishments and possibly reduce team morale.
Previous article: Release: Communication, Training, and Deployment
Continue on to the next article: The Retrospective: Working Together to Improve
Copyright
©2009 by Manning Publications Co. All rights reserved.
No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by means electronic, mechanical, photocopying, or otherwise, without prior written permission of the publisher.