Share via

Biztalk Performance Lab Delivery Guide - Performance Lab Process - The Scope Phase

  • Article

The Scope Phase

This section will include sub-sections that discuss considerations for conducting the Scope phase of the Performance Lab Process.

Perf Lab Process Scope Phase

 

Why Consider Scope?

A common mistake, all too often made, is to underestimate the scope for a Performance Lab Engagement. It is very unlikely that a team can build an entire production-like environment and completely test a complex production system in only a few weeks. Failing to realize this can doom a Performance Lab Engagement to failure before it even starts.

A team should pick battles carefully. In a Performance Lab only scheduled to last a few weeks at the most, the team should be laser focused on one or two, maybe three at most, key performance issues. The application to be tested should be specifically developed to focus on the key areas of focus and should factor out all other technology variables as much as possible.

Occasionally, the organization requesting the performance lab simply will not understand the performance characteristics of their BizTalk solution enough to provide input into the Performance Lab Engagement. In this scenario, the best approach is often to make the first engagement’s objective to understand the basic performance of the various parts of the solution. Additional Lab Engagements can be scheduled to go into more detail.

A Document Engagement Summary

After it has been determined that a Performance Lab Engagement should take place, it is critically important to document certain specific details about the engagement as quickly as possible. Ideally, this can be done as agenda items during the initial meeting or call. One reason is to ensure that all stakeholders agree on the Engagement Details. Another reason to get details in words is because many assumptions need to be made based upon specific details about hardware, software, goals, etc. A lack of communication or an assumption that doesn’t hold can cause ill will and possibly even cause a Lab Engagement to fail or be abandoned.

Later in this guide, there is a section about Engagement Documentation. Many of the steps and tasks in the Performance Lab Engagement Process involve creating content for various sections of the Performance Lab Engagement Document. Creating and updating sections of this document as the lab is conducted results in a much better a document than if the document is left as an afterthought.

During this step of the process, we want to complete content for all of the Executive Summary section of the Performance Lab Engagement Document, with the exception of the Summary of Results section:

Goals and Objectives Section of Document Engagement Summary

With all stakeholders involved, create from 1 to 5 unambiguous bullet points that clearly state the goals and objectives of the Performance Lab Engagement. These bullets will typically address one or more of the following topics:

· Throughput

o Ex. Achieve highest possible throughput with predefined solution constraints and optimal hardware

· Latency

· Peak Loads

· Time to Complete a batch of size x

· Recovery from floodgate situations

As a best practice, it is recommended that goals and objectives be stated in terms of “achive highest possible X given hardware and software contraints” as opposed to giving specific performance numbers as objectives.

It is also best to state exactly how goals and objectives will be measured. The best technique is to build the wording for how the goal is measured right into the goal itself. For example, “Maximum Sustainable Throughput of XX end-to-end as measured by the counter ‘XLang/s\orchestrations completed / second’”

Knowledge to Consider

EstablishingPerformance Criteria in MSDN

Terms and Concepts We Use in Patterns and Practices on Codeplex

How To: Identify Performance Testing Objectives

How To: Consolidate Various Types of Performance Requirements and Testing Objectives

How To: Coordinate Performance Testing With an Iteration Base Process

How To: Evaluate Systems to Improve Performance Testing Effectiveness

How To: Quantify End-User Response Time Goals

 

Roles and Responsibilities Section of Document Engagement Summary

Clearly documenting, at the start of the engagement, and obtaining agreement between all the stakeholders as to the specific roles and responsibilities of each party involved in the engagement is critical to success. Doing this step well can help a team avoid all manner of confusion and miscommunication.

Typical examples of Roles might include the following but each situation may require other roles:

· Customer

· Partners

· Consultants

· Software Vendors

· Hardware Vendors

· etc

Typical responsibilities might include the following, but each situation may require other responsibilities:

· Provide specific hardware

· Provide specific software

· Provide specific expertise

· Logistics

· Connectivity, especially if lab hardware is remote

· Provide Specific deliverables

o Application to be tested

o Automated build

o Automated Deploy

o Automated Load Tests

o Automated Load Generation

o Test Data

o Documentation

For examples of what might go into the Roles and Responsibility section, see the Engagement Documentation Section of this guide.

Engagement Timeline

A project plan or Gantt Chart is typically overkill for Performance Lab Engagements, at least in our experience, so we don’t create them. What is very helpful though is to create a timeline. Visio is especially good for this task using the Timeline template. An example of a timeline might appear as in the image below.

Example Perf Lab Engagement Timeline

d

It is important to keep the timeline simple enough to be easily understood yet detailed enough to communicate milestones effectively. The Milestones that noted on the Time for a specific Performance Lab Engagement will vary with circumstances and customer requirements. The actual Milestones should be agreed upon by all stakeholders and reviewed regularly. One technique, which isn’t depicted in the image above, is to use color coding to indicate who is responsible for delivering each milestone. Indicating which roles are responsible for which milestones can add addition communication value to the timeline.

Knowledge to Consider

Integrating Visio 2007 and Project 2007

Integrating Visio 2007 and SharePoint Products and Technologies

Hardware Diagram

This section will discuss considerations for creating an initial description of the hardware stack, communicating those details to all stakeholders, and an example Hardware Diagram.

Why Create a High-Level Hardware Diagram

During the Scope phase of the project, the intention is to define the project, discover risks, and explore the feasibility of the engagement. One factor that is almost always an issue is hardware availability! Since this is such a critical issue, spending some time during the Scope Phase seems like and obvious good-practice. Even when hardware is available, it typically requires some lead time to be provisioned, imaged and generally made ready for the engagement. Don’t leave the hardware details too late in the project!

Example High-Level Hardware Diagam

In the image, an example of a High-Level Infrastructure or Hardware Diagram can be seen from an actual Performance Lab. There are many and perhaps better ways to create these diagrams but this is what we run with and it seems to effectively communicate the hardware needs at this phase of the project. What’s most important is the information communicated not the art used.

Example Perf Lab High-level Hardware Diagram

Hardware Information to Consider in the Scope Phase

· Machine Architecture (x86, x64, IA64, etc)

· Processors ( Type, Speed, How Many, cores, use of Hyper-threading )

· RAM ( Amount each machine )

· Local Disk Storage (type, size, speed)

· SAN ( Storage Requirements, How many LUNS, SAN card type )

· Network Cards (number in each machine, 100MBit vs. 1000MBit )

· Are firewalls present

· Is Network Load Balancing Hardware to be used

· Are specific machines to be clustered?

Software Information is also Useful!

· Operating System for each machine ( 32 or 64 bit, clustered or not, etc )

· Server Software to be installed on each machine

· Virtualization used if any (not typically used for a Performance Lab but will be more common in the future )

· Specific software features not typically installed like

High Level Architecture Diagram

Why Create a High-Level Architecture Diagram

In the Scope Phase of the process, we are always trying to discover risks. Once the Goals and Objectives are defined, it’s a good idea to learn as much as possible about the details regarding the design of the system, the features and technologies used by the system, the complexity of the system, etc. Depending of the circumstances of a particular engagement, the software solution may have varying amounts of documentation. At the very least, obtain from the customer a two or three pager showing basic information about the solution the parts of the system, the features of BizTalk Server Being Implemented, what other software technologies are part of the solution, how the system is divided into parts, etc.

Example High-Level Arch Diagram

I dream of a day when we’ll create extensions to the Visual Studio Distributed System Designers to make that tool more optimal for precisely modeling BizTalk Server solutions. Unfortunately, that day has not yet come. You can still use VSTS for Architects to produce some pretty good diagrams to explain and communicate the design of the solution to be tested during a Performance Lab Engagement. This task is typically either done by the customer or the vendor that developed the solution, and the output varies for every project. Fortunately the output isn’t as important as the shared understanding between the stakeholders that many different formats, documents and diagrams can convey.

At some point, I will create an example, but all the ones I have at this point were confidential to the customer.

Comments