Share via


Thoughts on Testing - Breaking Down the Test Plan

Test plans, or test specs, are designed to share with a product team exactly what a tester plans on covering to ensure that their feature is ready to ship to customers. 

Problem 1: Who Is the Test Plan For?

Thinking about who the test plan is for can greatly influence the results of what comes out of the actual document. There are usually a bunch of vested parties such as developers, program managers, peer testers, test managers, test vendors, sustained engineering teams and security experts just to name a few. Each one is looking for something different out of the test plan document.

Here’s a breakdown of the lifetime of a test spec and who cares about at each milestone:

Milestone

Who Cares

What They Care About

Design Phase

Developers

Program Managers

Peer Testers

Test Managers

Testability of a feature

Design of a feature

Manual and automated test coverage

Impacts to other features

How long testing will take

Test Strategy

Implementation Phase

Developers

Program Managers

Test Managers

Test Vendors

Security Experts

Test Coverage

Checklist of signoff items

Security Scenarios

Test Strategy

Stabilization Phase

Test Managers

Program Managers

Peer Testers

Test Vendors

Sustained Engineering

Test Coverage

Checklist of signoff items

Test Strategy

 

Post Release

Sustained Engineering

Security Experts

Peer testers

Test Coverage

Test Strategy

Security Scenarios Checklist of signoff items

The two topics that are cared about in every milestone are test coverage and test strategy. Put simply, it is what is covered and why. Most of the time, test specs and test plans focus more on items in the design phase because this is when test plans are usually reviewed and iterated on. The people in the test plan reviews chose to focus on those items.

Problem 2: What Is the Test Plan Used For?

It isn’t always clear what a test plan is used for once it has been written. Often times, there are notes of work items and costs, lists of tests that need to be run or automated and perhaps specific performance and behavior goals to reference but it usually isn’t looked at again until sustained engineering handoff or something’s gone wrong.

Making test plans something that are consistently referenced, reviewed and used should be a goal for any test team. Otherwise, the exercise of writing the test plan is simply to think about how someone would test the feature and highlight areas of concern in a developer’s design.

Let’s go back to a subset test plan users:

Users

Needs

Feature Tester

Design Phase

Testability of a feature

Manual and automated test coverage

Test Strategy

Implementation Phase

Test coverage

Test work items

Test cases

Security scenarios

Stabilization Phase

Test Coverage

Test Cases

Checklist of signoff items

Test Managers Feature Team

Design Phase

How long testing will take

Test strategy

Test coverage

Implementation Phase

Test coverage

Test work items

Test Cases

Stabilization Phase

Test Coverage

Checklist of signoff items

Sustained Engineering Security Experts Future Feature Testers

Stabilization Phase

Test Strategy

Test Cases

Test Coverage

Checklist of signoff items

Problem 3: Keeping the Test Plan Up-to-Date

If developers are following more traditional design then implement models, the test plan will usually be signed off prior to a developer starting implementation work. What happens to the test plan now?

There are three scenarios to consider:

Scenario

Test Plan Needs Changes

Developer Design and Spec do not change, feature is implemented exactly as described

No

User requirements change and minor updates or design changes are needed. Feature goals modestly updated and changed

Yes

Major changes to requirements and design cause implementation to drastically change. Feature is implemented completely differently from original design.

Yes

But what usually happens in these situations? If deadlines or milestones are tight, the actual test plan might not be updated. Or, more often than not, the test cases and scenarios are updated on a case by case basis in automation but not necessarily captured in the original document.

Test Plan of the Future™: Features and Requirements

So, after all of that what does the Test Plan of the Future™ need? Here are some of the requirements that I thought of that will address the needs of the primary users:

  • Test Items
    • Test Cases
      • Manual
      • Automated
      • Security Specific Tests
    • Test Coverage
      • Code Coverage
      • Scenario Coverage
    • Checklist for Signing off
      • Items discussed among feature/product owners
    • General Test Strategy Notes
  • Feature Spec/Dev Items
    • Features
    • Prioritized Scenarios
  • Easy to Update
  • Reflects the current state of the product

This list of requirements satisfies a majority of the needs laid out above. If test plans can reflect the current state of the product, highlight feature and scenario coverage all while being easy to update, it can become a really valuable tester resource. 

Test Plans: Looking Ahead…

As things progress, I’m going to try investigating and implementing different types of Test Plans of the Future™.  Stay tuned for future posts on this topic…