Data Driven Quality https://docs.microsoft.com/archive/blogs/kenj/feed.xml Several of the older posts come from when I was writing on testing, SOA, and Testing in Production (TiP). I am currently writing on Data Driven Quality (DDQ). Tue, 16 Apr 2019 05:17:14 GMT en-US hourly 1 The future of quality is easy with EaaSy and MVQ https://docs.microsoft.com/archive/blogs/kenj/the-future-of-quality-is-easy-with-eaasy-and-mvq Mon, 19 May 2014 16:01:00 GMT https://blogs.msdn.microsoft.com/kenj/2014/05/19/the-future-of-quality-is-easy-with-eaasy-and-mvq/ Introduction to EaaSy and MVQ Last fall (2013), I published two papers internally to Microsoft. I wrote them to challenge the status quo and paint a vision of where I felt the future of software testing was headed. The papers have resonated across Microsoft and some elements have leaked out to the Internet. The first paper, and the more heretical, was about Minimum Viable Quality (MVQ) and the other was on my personal perspective that every software and even devices project should be developed like a service. This second concept I call Everything as a Service “yes!” (EaaSy). The “yes!” is there to both make the acronym easier to remember but also to signify that it’s about time. This is my first public post on the concepts of MVQ and EaaSy. I did present the concepts in a talk at the ALM Forum in Seattle April of 2014 and the slides are currently available on the event site. There are several other individuals writing about moving to lean data driven quality. All of these bloggers I have worked with and shared ideas back and forth. Brent Jensen wrote a post where he commented on MVQ in March of 2014. In his post however he was focusing not just on MVQ but also on “Shifting the Tester Mindset.” Alan Page focuses on a similar aspect of tester culture with his calls for testers to stop writing automation. See Alan’s May 2014 post, “Stop If You Want To.” Seth Eliot highlights rich fast data and the many uses of it in his writings and speeches on Data Driven Quality (DDQ) and Testing in Production (TiP). DDQ goes much deeper into all the ways that data is changing all approaches to testing and quality. Seth and I have plans to write more on the topic of DDQ later this summer. Those are all great posts that I recommend you also read. So, let’s dive in. Minimum Viable Quality (MVQ) was born out of my frustration at being a speed bump. Early in my career I thought my job was to stop bad bits from getting to end users. I was there to stand in the gap and when an evil senior manager wanted to ship a product that wasn’t ready for prime-time. I would raise my staff over my head, strike the tip of it on the ground and yell, “You shall not pass!” Down would go the evil Balrog along with the bad code. Or at least that’s what I thought would happen. What really happened was that despite having great test results showing the problems with the code, quite often the decision was made to ship anyway. Well, after getting run over a few times and squashed flat like an ice-cream cone dropped onto a busy freeway, I realized my job wasn’t to stand in the gap, but rather to help the team figure out how we could ship the product on time with good enough quality. MVQ is just another way of saying the quality of the product is good enough for the target audience and well, no more than that either. You could consider MVQ to be lean testing. Just enough testing and no more than is needed at that point in time. EaaSy is actually a framework that stems from my fifteen plus years developing and shipping online services and applying those lessons to developing and delivering all software products and even devices. To really adopt an MVQ approach to quality you need to implement the EaaSy framework. With that in mind I’ll introduce EaaSy and MVQ and then dive into each in more detail.

EaaSy and MVQ are the future of software quality

In the future I don’t think we will call what we do to ensure software is of the right quality “software testing.” As we trend toward everything as a service, quality will shift toward being a data driven service itself. What are these trends that are driving us toward EaaSy and MVQ? Well they all stem from a set of very dominant industry trends.
  • First and foremost the business model for software has changed. With the cloud people really won’t be buying software much going forward. Not even an operating system will be purchased. Consumers and businesses are buying services for communication, storage, data processing and many more. Consumers do buy Apps when they want or need them or want to enjoy a game but those apps are serviced through an online store and even in that space the revenue model is trending toward in-app purchases and ads. Consumers and businesses devices that come with an operating system that they expect to be updated with bug fixes and cool newness on a fairly regular basis. People don’t stand in line for the next release of Windows but they do stand in line for the cool new device that has a friendly operating system and cool apps. So the days of standing in line at a big box retailer for software are gone and so the business model for software is changing.
  • The large and competitive consumer technology market has demonstrated that users demand more rapid innovation and are willing to purchase new technology at a rate far greater than just a few years ago. That means there is more demand and more expectation for rapid innovation. They have also demonstrated a greater tolerance for minor regression, especially if they are fixed quickly.
  • All across Microsoft and the rest of the software industry we are seeing teams increase their rate of release. Not all are doing this with a services model in mind but the mere fact that we ship more frequently for devices, apps, operating systems and even server products, everything looks and behaves a lot more like a service.
  • Everything is now connected and thus updatable. Whether it is a new “smart” TV or even my FitBit, smart things are eventually connected to the internet and thus all software on them is updatable with some level of reliability. Some sectors like critical medical devices will not be on this rapid update pace but that is simply early adopters vs business segments that due to compliance and other factors must lag.
The market dynamics mean that we must think of new ways of shipping software at a lower cost, ship it more frequently, and maintain an acceptable level of quality. To help frame that discussion I created EaaSy as the foundational capabilities for enabling this new faster iteration cycle and then created MVQ as a way of aligning the goals of the quality engineers with the new business imperatives.

The EaaSy 5 key capabilities

EaaSy is my framework for communicating a set of five key capabilities (see Figure 1) required to allow just about any product to look like and be managed like it was a service. They are all geared around reducing risk when developing and releasing software products. Managing Risk is essential when building complex products.

Ken aside – It occurred to me some number of years ago that my role in test was less about finding and fixing bugs but more about producing a signal for management to better understand risk to quality. Think about it, if you have a complex product that has not yet been released to the public you will have very little data available to you for assessing if the product is of acceptable quality. Test results are a proxy for real information. Of course with Testing in Production we now have real data so test results are becoming a bit less valued.

When in place, they allow any software engineer to ship faster with well-reasoned and managed risk in terms of catastrophic bugs impacting large numbers of users. In fact, as the math will bear out, it is safer to ship faster with EaaSy in place than it is to invest in significant up front testing.

Figure 1: there are five key capabilities required to adopt EaaSy.

There are five key capabilities required to successfully implement EaaSy and to make your product look like and behave like a service. Those five capabilities are:
  1. Componentization allows the product to ship in independent modules
  2. Continuous Delivery of small changes by components
  3. User Segmentation based upon risk tolerance
  4. Flighting both for evaluating quality and for experimentation
  5. Rich fast data for Quality of Service (QoS) and Quality of Experience (QoE) analysis
With these capabilities in place a product should be able to be updated with new code on a daily basis. Now getting all these capabilities in place on your product team won’t be easy and they won’t happen overnight but when you arrive, you will be able to ship updates much more easily than ever before.

EaaSy Capability #1: Componentization

I want to start diving into EaaSy at the foundation. To me, componentization is the foundation of moving into a services architecture. One thing we learned early on in the services world was that you couldn’t stand up all the pieces of a large scale service all at the same time. This forced us to define and maintain contracts between service components. The canonical example of this is authentication. Most online services today do not run their own authentication service but instead trust other cloud services such as Twitter, Facebook, Google, and Microsoft. Figure 2: Example of common components (sometimes called layers) of a service In the software, apps, and devices world it is common to think in terms of getting something on a retail store shelf. That leads toward a philosophy of sim-shipping all components at once and tends to break down component boundaries. The product ships as one unit so it should just all work together as one SKU. Componentization and backward compatibility are often lost in this release model.

Ken aside – Every time I think of sim-shipping I think of the Titanic. I think of a massively complex product with hundreds of passengers all betting that the voyage will be a fabulous success. The problem is that when that ship hits an iceberg, the entire product is now in jeopardy. Things start to flood, fall apart, the ship (a metaphor for our software product) is sinking. Ship together and sink together as one.

The antithesis of sim-ship is to launch a hundred self-sustaining little boats. Together those boats can get all the passengers across the ocean with much less total risk. If one of them or even a dozen hits an iceberg and sinks the relative impact to the overall customer experience is less than the big ship going down. Now I’m not saying a hundred little boats is the same experience as a big grand ship like the Titanic. No, I’m using this as a metaphor for relative risk. From a pure engineering perspective it would be less risky to have more boats than fewer.

Think about it.

From a tester perspective componentization is nothing more than problem decomposition, something we do all the time. I see a bug, let me figure out where the root of the bug is by isolating factors. The most common type of bug I find in a service is a regression and those are caused by some sort of change either in the code or the environment in which the code is running. With a regression we often focus on what changed when and how that led to the bug. Problem decomposition. With services when we would ship updates to any one of the service layers we would need to ensure we were both forward compatible with the new features and that we maintained backward compatibility. Each service component could be updated independently and the integration points as well as the end to end experience validated. Componentization and asynchronous releasing is actually easier than trying to get one version of all the parts of a product to work relatively bug free in a single release. The reason it is easier is because componentization that is reinforced with asynchronous release processes teaches us to design better for forward and backward compatibility and to code for it up front. Risk reduction become built into the engineering processes.

EaaSy Capability #2: Continuous Delivery (CD)

Continuous Delivery (CD) is quite simply less risky and it leads to better componentization. Let me start with the impact CD has on componentization and then get back to the risk part. While componentization sounds great, I have rarely seen it work unless it was combined with Continuous Delivery. The reason for this, I think, is simply repetition. The more experience a team can gain with a new practice the stronger and quicker it will grow. When I was working on MSN in the late 90s we would aim to release the MSN client CDs once a year just before back to school season. AOL followed a similar model. Throughout the year the different service component would work on their features and continually validate they were working in a test environment we called the Integrated Test Environment (INT). Near the end of the yearlong cycle we all started upgrading the production services. We did this in pieces but we kept breaking each other. The reason for this, in my opinion, was that while we created contracts between the layers, we engineered and tested as if we were one cohesive unit. The INT environment was actually our downfall for componentization. It wasn’t until years later while working on Bing that I was able to observe the role continuous delivery with independent ship decision had on the success of componentization. Let me go back to the risk reduction side of CD. From an Operations Engineer or a Test Engineer’s perspective change is synonymous with risk. If I get a brand new PC, take it out of the box, and turn it on, it pretty much always works. Over time new patches get applied, add-ware sneaks on and before you know it I’m better off reinstalling the operating system from scratch and starting over. Change leads to bugs. The same is true in the services world but the impact of a bad change on a large scale service can be catastrophic.

Ken aside – It’s not good to yell at your engineers when things go wrong. It creates backlash behavior that can be even worse than the original error. I have had to go through two major un-learnings over the past few years and it occurred to me that both of them were results of being unduly harangued for a problem in production.

The first bad decision was helping design the INT test environment. The test managers of MSN at the time felt it was possible to mimic production well enough in a lab that we could catch all the critical bugs before we shipped. INT was directly born out of the frustration of test managers being blamed when a bug would get into production.

The second unlearning comes from my experiences in operations. Ops engineers, more than any other type of engineer, know that change leads to issues. That is why operations is so focused on changed management processes and documentation. Why is this? Well when a bad change goes in, everyone turns to ops and asks how they let this happen. The change management system is a way to track and assign blame.

The problem with both over testing in INT and excessive change management is that they slow down innovation and do not prevent bad bugs from getting through. They are counter to continuous delivery and don’t accomplish what they were designed to do.

It seems counter intuitive to say that shipping more frequently reduces risk but it does. More code into producing more quickly means less code churn per release. Changes are smaller and thus less risky. Additionally CD promotes componentization.

EaaSy Capability #3: User Segmentation

While I believe all five EaaSy capabilities are required to adopt MVQ I have to admit that user segmentation is the most important of the five. The reason for this is simply around managing risk. If you cannot divide the users of your product up into different segments and expose new code and features by user segment then by definition, when you first release your code, you must hit the release ready quality bar.  When properly implemented, the great thing about user segmentation is that you can now create pools of users with different characteristics such as heavy or light users and you can create risk pools. In Microsoft many teams are adopting the rings of risk model (see Figure 3). The rings signify that at each layer the number of users actively using the product is increasing and the tolerance for defects is decreasing. A typical model would have four or more pools of users divided across the rings. Figure 3: Code Quality increases as the code is exposed to different types of users with ever increasing expectations of higher quality. At the inner most ring you have the feature development team itself. These groups are actively working on the product and it’s not uncommon to even try out a buddy build from a team member. Even within Windows and Office teams do this regularly. The next ring might be a sizable product team. Think of a team say developing the Office Outlook client to say a team the size of Windows. The team as a whole is pretty tight nit and committed to shipping a great product with great final quality. There should be a high tolerance for bugs, in fact excitement to find them within the team before release, as long as there are not so many defects that productivity becomes completely blocked. When you go to the next ring of an entire company the quality needs to be pretty good. These individuals may be committed to building the company but that doesn’t mean they will be excited to lose their work or have to try something two or three times before it works. Still, they will understand that finding bugs in house is better than releasing them on customers. At the edge between the company and a public beta is a special divider. This is what we call the disclosure line. When you start to bring beta users onto a product you have to expect leaks so you will need to have a communication plan in place. Additionally if your company wants to protect its intellectual property, in most cases those patents will need to be filed before the features are shared with the beta user pool. External public beta users are in many ways the most exciting ring of all. They are typically enthusiasts that understand they will be getting a pre-released version of a product that will likely have bugs. They are willing to take on some of this pain for the chance to be an early adopter. Perhaps the most exciting aspect of beta users is that they are not typically engineers and will likely use the product in ways none of the smaller rings every thought of trying. Going down new code paths helps discover new bugs. The final ring is everyone that may be interested in your technology product. At this point you should be trying to monetize your product and individuals using it will expect that it is of good quality. That is true for products they purchase and also for free products that are perhaps monetized through ads. Many companies use similar models to manage how they release features. Google is known for running hundreds of experiments a day on their search portal. Gmail was labeled a beta for years. When Facebook rolled out their new timeline UX early adopters could opt into the new experience. Similarly when Microsoft re-launched Hotmail as Outlook.com users had a very long period to opt into the new look. In the AppStore model you are also seeing the adoption of user segmentation. These features for identifying and managing pools of users for Alpha and Beta releases are becoming common place in all the major AppStores. Image 1: Article from http://support.google.com/googleplay touting the beta and alpha features of the Google Play store platform. User segmentation isn’t just for services. There are countless examples of products shipping on CDs and DVDs that have leveraged user segmentation. There have been Beta releases of Windows since before the very first version was ever released. This method of segmentation was usually fairly manual and often spread out over long periods of time. The difference with EaaSy is that we are now taking products like Office and Windows and XBOX and building and releasing them like services. This requires a fully automated way of managing user pools and the versions of the code they receive.

EaaSy Capability #4: Flighting

Inside of Microsoft we use a term Flighting to mean something similar to the industry terms experimentation and A/B Testing and the associated capability of turning code paths on and off via configuration settings in a cloud service. In this case flighting also integrates with user segmentation (EaaSy capability 4) such that we can ship code to users within a particular ring and then configure the new code on and off at a later time. Some groups inside Microsoft call this shipping dark. The code is out there but it is not turned on until the configuration change is made in the cloud. All of these capabilities come from lessons learned from A/B testing. In the case of A/B testing it is common to have the ability to dynamically route users to different experiences or different code paths. This is typically done to experiment on a UX treatment to determine which one has better engagement with end users. Image 2: Example of A/B testing routing users to different code paths. Image from “The Ultimate Guide to A/B Testing” by Paras Chopra Examples of A/B tests for a service might be a new treatment for where Ads are placed on a portal page. The goal of the experiment would be to answer the question of whether or not users will click more often or if the page is now so obnoxious users will stop using the service. With cloud based flags, users are routed to the control and the experiment. These two treatments are different code paths living side by side in production. With flighting this same dynamic routing capability is leveraged by quality to roll out new code in the dark state and then have real users hit the new code path. In this way you can control the number of users hitting a new code path while answering the basic question of whether or not the new release is good enough or too buggy. This technique of flighting as a way to manage new code exposure to real users is a variation on what services call rolling upgrade. The difference here is that access to the new code is managed via a configuration. This improves rollback because removing the new code from production use is simply an undo on the configuration management service. This technique is being used by all types of technology. Apps acquired in app stores are actively using this technique. In the following article snipped I was reading how Facebook developed and released their iOS 7 app through the Apple AppStore but they managed to implement their own ability to flight and test different UX experiences (see references). It seems they value the capabilities they have with their online presence so much that they built those capabilities into the app and supported it from their own service infrastructure.

How Facebook secretly redesigned its iPhone app with your help

To get around these issues, Facebook built an entire system for creating alternate versions of the native app within the native app. The team could then turn on certain new features for a subset of its users, directly, and measure the results. Starting in early 2013, the team put together a system of “different types of Legos — that we can reconfigure really easily … and see the results on the server in real time.”

From article on The Verge by Dieter Bohn September 18, 2013.

EaaSy Capability #5: Rich-Fast-Data (RFD)

One of the things we discovered early on in the world of services is that data is the life blood of quality. With services the ability to get telemetry off of the servers vastly increased our ability to detect problems and fix them. We also learned how to mine data for a better understanding of how users were trying to interact with features. I’m referring to this as Rich-Fast-Data (RFD) because to really have your product work like a service you need a lot of good quality data and you need some of it in near real time. With RFD all sorts of systems can be put in place to automate risk mitigation. RFD works with user segmentation, with componentization, and with Flighting and the configuration managements systems. It is the final element to enable a team to move fast and safely with EaaSy.

Ken aside – User behavior and engagement are the ultimate arbiter of quality. If you ship new code and your monitors and your telemetry tell you everything is fine but you see a statistically significant negative change in user interaction (usually meaning usage tanks) then you’ve likely shipped too many bugs. These bugs may be code defects or they may simply be basic usability but either way something is wrong. If users don’t like something they will give you less of what you want and that is their time.

The real key to success in adopting a RFD approach is how the team leverages the data to mitigate risk, quickly respond to defects in the wild, and drive continuous improvement. One thing I look at with respect to a bug in the wild is an old service metric called Time to Detect (TtD). Quite simply if TtD is high then that means there is a gap in monitoring and or instrumentation. Remember MvQ is mostly about reasonable risk in terms of user impact.  If a defect is introduced and even if it is small but the time to detect is high, then more people experience the defect.  That is symptomatic of a weakness in your RFD strategy.  You might be fast but you are not complete enough.  When you do ship that big critical bug and your TtD is long then you have a brand damaging level defect.  Now there will always be gaps in telemetry so you will need a RFD proxy that will help with roll back decisions even when the telemetry says everything is good.  I was literally in a meeting today (10/31/2016) where a team from Office was explaining how they use net star rating on any update to any app in the iOS, Windows, or Android app store as a roll back signal.  If the update goes out and the stars drop then the release is bad regardless.  One time I was working on star rating in the Windows App store and NetFlix had an outage.  Just for fun I took a look at the star rating of the NetFlix app over the time period of the service outage.  Guess what, the stars plummeted.  In the comments it was clear many individuals knew this was a service outage but they used the star rating as a way to send a signal to NetFlix.  The outage ended and several individuals changed their star rating back up.

Where MVQ came from

MVQ really builds on other lean development practices. It is in a way Lean Testing. It’s an area I’ve been thinking on for a few years now as I have seen Big Data start to enable all sorts of new capabilities across many different industries and job roles. Back in 2012 I gave a talk on software testing and the move to Big Data with my Dev Manager peer Reena Agarwal. In the talk we covered how the rise of Big Data was beginning to impact the techniques of software testing of online services. I hadn’t thought of EaaSy yet so I was confining my thinking to the online space. During the Q&A I stirred up a bit of controversy by stating that I felt we were simply testing services far too much before releasing them to production.  This thought was bubbling up in me because I’d been reading “The Lean Startup” and wondering how the concept of Minimum Viable Product (MVP) could be applied to my work at Microsoft. Several attendees of the event were incensed by my suggestion to test less. It was a testing conference after all. I was even accused of suggesting we ship “crappy code.” Nothing could be further from the truth. MVQ is about shipping great software with the right level of quality and amazing features as quickly as possible at the lowest possible cost so that you can charge less and gain more users. In many ways MVQ is a lot like lean manufacturing but for software. I’m sorry but testing, especially over testing, adds cost and drives up prices. MVQ can be applied to all software and device projects and does not require MVP.  I want to recognize that this thinking on Lean QA was prompted by “The Lean Startup.”  It should also be noted that many others have written about just enough testing, just in time testing, and test driven development (TDD).  All of that work has influenced MVQ.

Ken aside – Satya Nadella was talking to us a few years back and he said that we “needed to be a learning organization.” In the Bing test org, this was often brought up as fail fast. The idea has its roots in industry but it became part of the lexicon of the original Live Search (pre-Bing name) test org from its earliest days and it helped drive a mind-shift change that is very powerful for any development or test org to embrace. I didn’t initially see the connection between fail fast and learning organization but now I would summarize it this way. When you can try fast with low friction, when you can discover fast through real data, and when you can take all of that and build up knowledge on how to succeed in small and big ways fast, you can become a learning organization. You learn fast!

The only thing is, you must accept that you will fail, a lot, along the way. You will have to learn what good failure is and how to learn and grow from smart failures. I wouldn’t get too hung up on the quality of the failures because if you fail fast you will not have much sunk cost into any one failure. That’s a key element to fail fast. You minimize your losses and move onto the next grand idea.

“Fail Fast” and its corollary “Learn Fast” are much deeper than even what I just summarized but that framing has helped ground me into “Learning Organization.”

What is MVQ

It’s funny, I’ve posted and talked many times now about the EaaSy capabilities and MVQ and I’m always getting asked, now what exactly is MVQ and how do I do it? Here let me just walk through the possible levels of testing: Under Tested – that means your product is below acceptable levels of quality. You will know this when you try to get individuals to user your product and they stop using it. Over Tested – At this end of the spectrum you will have massive test labs and massive suites of tests. You likely have a long stabilization phase and you have multiple week long test passes. Here’s the rub though. Of all of those tests, manual and automated, most of them find nothing wrong with the product. I would go so far as to submit that every time a test runs and doesn’t find a bug, that is wasted effort. Figure 2: MVQ focuses on less up front testing and more rapid releases to production. MVQ Limited Release – This is your beta, your friends and family, you tech enthusiasts release. We all know that we will allow more bugs in a limited release than in the final release. In the MVQ model we would apply the EaaSy capabilities to segment this limited release user base into smaller groups and leverage automated deployment and fast roll back to test even less than normal and push bits out to the users. MVQ Final Release – Okay, here we must meet customer expectations with quality. We can’t miss the mark here or we will get negative reviews in social media and the product will fail. Even with that in mind, we don’t want to waste time and resources over testing. This is where the Rich Fast Data really helps us. Even on a V1 product, let’s say a new gaming console, you can ship with known bugs. It takes time to get that product into the channels and into the retail stores. If the device is connected then you still have time to find and fix bugs between the time it is released and users start to turn the device on. Have you recently purchased a device that made updates before you could really use it? Hmm, maybe they found bugs and fixed them.

MVQ = Minimum Viable Quality and that means that as a tester you look for every possible excuse to ship the product as quickly as possible with acceptable quality and reasonable risk.

Risk, that’s the key isn’t it? Reasonable risk. In the old days with CDs and the high cost of recall class bugs, reasonable risk meant pretty darn solid quality. Also with the lack of connected users there were long delays in hearing about a bug in the wild. For those reasons we had to rely upon massive suites of tests run in labs. Not now, not in todays connected devices world.

And you’re never done shipping

What I learned from being in the services world is that you never actually ship a service.  There is just the next feature, the next scenario, the next set of performance improvements, and the never ending list of bugs to fix.  The concept of shipping comes from a time when product were developed over two and three year cycles and release on a CD. Shipping doesn’t apply in the modern world whether it’s an app, a device, an operating system, a server product, financial software, social networks and any bit of technology. If it is connected it is updatable and if it is updatable, it’s never done shipping, at least until you are ready to retire it.

Ken aside – If you release a version of your product to a set of target users and you have to roll back three times in a row then your quality bar is probably too low.  If say one out of ten daily releases has to be rolled back, then your quality bar may be just fine.    

What is the advantage of taking an MVQ approach to software product delivery?  The bottom line is that you start to get data about how the code is functioning in production with real users more quickly.  The key aspect is to balance the minimum such that the feature set and the quality of those features within the product is not so low that the product doesn’t get used. If users aren’t using the product, no matter what segment/pool they are in, then you don’t get data. If data is too low, then you won’t discover and learn the harder to find bugs because the code won’t be exercised.

Ken aside – I would rather miss a recall class bug that gets into production with minimal real end user impact than hold a release for an extra day to find and fix three bugs that would never actually manifest in production.  While that statement may sound cavalier, it has quite a few qualifiers so don’t think I’m totally flipping the bit on quality.    

The great thing about MVQ is that you can more quickly get into the final phase of real integration of your software, services, and devices and real usage by real users. When MVQ is implemented correctly with EaaSy, it is not only more efficient, but also less risky.

Final Thoughts

To get to the world of less testing, shipping faster and hitting just the right level of quality you need to implement the five key capabilities of EaaSy and then change your mind set to focus on Minimum Viable Quality. The best path to EaaSy that I have seen is to allow teams to ship their components on independent schedules. That would promote both the concept of componentization and to a lesser degree continuous delivery. The reason this works is simple. Developers like to ship. If you let them ship faster and on an independent schedule, they will figure out how to maintain forward and backward compatibility. That is the key to componentization and the first foundational step toward making your product more like a service. The capabilities of user segmentation and flighting improve the control you will have on the levers of risk. If you can roll back bad code quickly, pushing a bug to a small set of users becomes reasonable risk. The last piece is rich fast data. Some companies and even cloud platforms make this available to product developers natively. Even with the capability to have rich fast data in place, developers often under instrument their code. That is, until there is a catastrophic failure in the product and they do not have the ability to solve the issue quickly. Yes, the path to MVQ almost always includes disaster. The reality is that all the top companies are moving in this direction. MVQ allows for lower cost and shipping more features to more users and delighting them than more traditional methods. Research and References: The Lean Startup: How Today’s Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses (book 2011 by Eric Ries) The Ultimate Guide to Minimum Viable Products (blog 2013 by Vladimir Blagojevic): Mostly provides an overview of key approaches to MVP from Lean Startup How To Create A Minimum Viable Product (article 2013 on Tech Crunch by Emre Sokullu): Article is mostly a list of open source technologies and SOA design ideas for launch an online MVP based upon Emre’s experiences launching Grou.ps In Pursuit of Quality: Shifting the Tester mindset (blog 2014 by Brent Jensen): Wake up testers, “fixing correctness issues on a piece of code that no one is using is a waste of time & resources.” Stop, if You Want to… (Blog 2014 by Alan Page): Testers should stop writing so much automation. The world of data is changing how we can approach testing and quality. Testing in Production, Your Key to Engaging Customers (presentation 2011 by Seth Eliot at STP Con): Scenarios and Testing in Production produce High Quality Product Delighted Customers. Create Your Roadmap to Data Driven Quality (presentation 2014 by Seth Eliot at STP Con): Ditch the HiPPO (Highest Paid Person’s Opinion) and shift to data from real users to gain real insights on quality. Good Enough Quality: Beyond the Buzzwords (IEEE paper 1997 by James Bach): Paper and article on reducing up front testing of product code. Journey to Continuous Deployment (presentation 2009 by Nathan Dye): I couldn’t find the white paper that I originally read by Nathan but he covers all the key points in this presentation.]]> Penultimate Day of School 2011 https://docs.microsoft.com/archive/blogs/kenj/penultimate-day-of-school-2011 Tue, 14 Jun 2011 12:18:00 GMT https://blogs.msdn.microsoft.com/kenj/2011/06/14/penultimate-day-of-school-2011/ If it’s June it must be time for my annual “I love the Penultimate Day of School” blog post.  As this is now my third post on the word penultimate (actually the second post was on the antepenultimate month of school).   In a sense that now makes my first post on the subject the antepenultimate post in the series.   If you feel a bit lost, don’t worry, I’ll clear things up right away. 

You see it all started about four years ago when we were approaching the next to last day of the school year for my children.  In grade school it seems the last or ultimate day of school is usually a half day and is mostly filled with cleaning up, packing up, and a wee bit of partying classroom style. 

That just didn’t seem right to me and my wife so we decided to have some fun the day before the last day (a.k.a. penultimate) of school.  To that end we taught our children the definition of penultimate, made a Facebook page, and sent the children of to test the vocabulary of their teachers. 

 

Let’s face it, penultimate is a very uncommon word and not very many people know the word, let alone use it.  I personally think it’s a brilliant word.  Let me provide a few examples of just how useful a word it can be.

The sports world is a great place to apply the word penultimate.  Most basketball enthusiasts will by now know that the Dallas Mavericks are the ultimate champions of the NBA for 2011.  That of course makes the Miami Heat the penultimate champions (newspaper ads aside).  It sounds like quite an accomplishment however I’m quite certain Miami fans are not pleased with this result.  It will be another day and a game 7 before we will know the winner of the NHL playoffs.

Penultimate also works well in the business world.  If you happen to work on a web search engine like I do, you will pay attention to the monthly comScore reports on query share.  If you go off the absolute rankings you would end up with the table below

Total Explicit Core Search

May-11

Rank

Google

65.5%

Ultimate

Yahoo!

15.9%

Penultimate

Bing

14.1%

Antepenultimate

Ask Network

3.0%

Preantepenultimate

Now Bing does power all the searches for Bing and Yahoo so in my mind that of course puts Bing in the penultimate position.  Of course we are aiming to be the world’s ultimate search engine but that will come over time and with executions. 

Arts and entertainment is a category where they don’t even like to use the phrase winner.  Take the Academy Awards for instance.  In that environment when they announce the winner the simply say, “and the Oscar goes to…”  Of course this phrasing was chosen specifically to not bruise any of the inflated Hollywood egos.  For me this is quite frustrating because we don’t have an actual winner and in the case of this year’s best picture category we have quite a few penultimate winners.  Let’s see exactly how many there were, Black Swan, The Fighter, The Kids Are All Right, Inception (my favorite), 127 Hours, The Social Network, Toy Story 3 (really?), True Grit, and Winter’s Bone.  I mean “The King’s Speech,” did win, er had the Oscar “go” to it, so that would leave us with nine penultimate winners.  Well clearly art is not a place where the word penultimate can easily be used. 

The spirit of the Penultimate Day of School is simply to encourage young minds have fun with big words.  There are many other great words out there that children can have fun with and in many cases stump the adults around them.  I quickly Binged word of the day and came up with a few sites and several great words.

Lest I too begin to bloviate I will end by simply saying, “Happy Penultimate Day of School.”

Ken Johnston @RKJohnston

Bing Domains Test Manager

External: Twitter | LinkedIn | You Tube | School Board

Author: How We Test Software at Microsoft (with Alan Page and BJ Rollison)

Available through Amazon.com

]]> Happy Antepenultimate Month of School https://docs.microsoft.com/archive/blogs/kenj/happy-antepenultimate-month-of-school Fri, 30 Apr 2010 09:37:50 GMT https://blogs.msdn.microsoft.com/kenj/2010/04/30/happy-antepenultimate-month-of-school/ “I love the word Penultimate,” and I love our family’s made up Internet holiday “The Penultimate Day of School.”  As we are now nearing the end of the antepenultimate month of school I felt it was time to make another post promoting our new holiday and highlighting my favorite family of words that now includes, ultimate, penultimate, and antepenultimate.  I could of course add the word preantepenultimate to the list but I think  I’ll save that one for a later blog post. 

Before I go into what has happened with respect to our Internet holiday over the past year I should probably explain exactly what the antepenultimate month of school is.  Well, in the United States the official end of the school year is almost always in the month of June.  Antepenultimate quite simply means, “third from last in a series.”  That would make April, this month, the antepenultimate month of school.  This is a very simple example so perhaps I can provide a few more interesting and humorous examples of the use of antepenultimate.

I thought of using an example drawn from the Miss Teen USA 2007 competition but I couldn’t do that without bringing up the case of Miss Catlin Upton and her disastrous response to the question in the final round. 

"I personally believe that U.S. Americans are unable to do so because, uh, some people out there in our nation don't have maps and, uh, I believe that our, uh, education like such as in South Africa and, uh, the Iraq, everywhere like such as, and, I believe that they should, our education over here in the U.S. should help the U.S., uh, or, uh, should help South Africa and should help the Iraq and the Asian countries, so we will be able to build up our future, for our children".[8][9

Miss Upton has suffered through more than 50 million views on YouTube but has also leveraged that faux pas into a dozen or more TV appearances.  It would certainly be a humorous example, but there is a problem.  Miss Upton ended up being the preantepenultimate finalist, and I promised I wouldn’t go into that word in this posting.

Instead let’s go with the Miss America 2010 finalists.  Miss America still maintains a core focus on scholarship assistance for young women.  I expect all of their finalists listened to and adequately responded to their questions. 

Contestant Final Result Penult Position
Caressa Cameron – Virginia Virginia Miss America 2010 Ultimate Miss America 2010
Kristy Cavinder – California California 1st runner-up Penultimate Miss Amarica 2010
Stefanie Wittler – Tennessee Tennessee 2nd runner-up Antepenultimate Miss America 2010

See, the second runner up is actually the antepenultimate winner.  It couldn’t get any easier than that, could it?

What else do we have that falls into the realm of antepenultimate things?  Let’s start by looking at other competitive endeavors.  In this case the focus remains on winning and so antepenultimate winners would be in third best.  We’ll get to third worst losers in just a bit. 

 

  • In the world of Dancing with the Stars Donny Osmond and his partner Kym Johnson were the ultimate or the final winners of season 9. Unfortunately for his sister Marie Osmond and her partner Jonathan Roberts they finished season 7 as the antepenultimate winners.  For the current season my guess is that Erin Andrews and Maksim Chmerkovskiy will take the antepenultimate spot.  And, yes I do watch the show but not of my own free will.  My wife likes the show and even my kids really enjoy it because they get to parade around the living room pretending to do tango’s and jives along with the stars on the TV. 

And those that didn’t really quite lose are?

Personally I feel focusing just on third place winners goes against the real intent behind the word antepenultimate.  I mean the definition is third from last in a long series so instead let’s now turn our attention to those finishing third to last. 

  • Of the 43 cars that qualified for the 2010 Daytona 500 Joe Nemechek was the antepenultimate finisher.  Unfortunately it seems that his race ended abruptly with a spin out and a wreck.  From what I found Joe is a well liked racer and was voted most popular driver in 1992 & 1993. at least we can say that in this case the nice guy didn’t finish last. 

 

  • In the case of politics it is rare to have more than two candidates on the ballot.  In the 2010 Florida US Senate Race there are three finalists.  According to a recent Rasmussen Poll, if the election were held today the finishing order would likely be Marco Rubio the presumptive Republican candidate with 37%, current Florida Governor and brand new Independent Charlie Crist is polling at 30% and 22% for the likely Democratic nominee Kendrick Meek.  In this odd situation that would mean Marco Rubio be both the antepenultimate loser and at the same time the ultimate winner.  Who said politics was not a cerebral pursuit. 

Penultimate Day of School is about fun with words

This blog pose however is not about racing, politics, sports or even Dancing with Stars.  This blog post is meant as a celebration of really gargantuan fandangled words and getting very young children to use them. 

Why the focus on the penultimate day of school you may ask?  Quite simply to give my kids and their friends something to do on the second longest day of any school year while secretly helping prepare them for future success. 

Let’s face it, the last day of any school year is often filled with hugs, exchanges of email addresses and phone numbers, and for the younger kids they often have a class party.  The next to last day, the penultimate day, teachers often force themselves to keep teaching.  Outside the weather is warm, summer is breaking out, many public pools are opening up.  The children can’t be expected to focus and so I submit that to a child the penultimate day of school must seem like the longest day of the year.

So, why not create a fun new made up holiday that helps children prepare for their future college entrance exams and life in the business world.  Just search for books on improving one’s vocabulary and dozens of resources pop up.  Vocabulary is big business. 

image

It really works.  Just a few weeks back the weather was getting warmer and my son decided we should play a little wiffleball baseball in front of the garage.  We used a pitching machine and you could keep hitting as long as you didn’t get three outs.  Of course this meant the roof had to be in play and we had to use a lot of ghost runners.  The problem for me was that I could never tag the ghost runners out, they always seemed to be faster than me.

After my son and his ghost runners outscored me about 227 to 10, long after the sun had dipped below the horizon and the sky had faded from bright orange to dark purple, and long after the outside photoelectric sensor turned the lights on, I decided it was time to go inside.  If you have little ones, you know that when Dad says it’s time to do something else, they know immediately there is still time to do just a bit more of whatever is going on at the moment. 

“Come on son, it’s time to take a bath and head off to bed,” I said.

“No Dad, you haven’t been up in a long time,” he replied. 

“Okay, how about this,” I said back to him, “what if you take one more at bat and then I’ll take the last four balls from the pitching machine?”

“Yeah! That means I’ll have the penultimate at bat and you’ll be ultimate,” he said casually.  Well you can imagine that I was quite floored that my second grader would use a word like that in casual conversation.

What to do with your child on the penultimate day of school

  1. Join the Penultimate Day of School Club page on Facebook
  2. Post a club event with the date of your schools PDofS.
  3. Pick a group of words to use with the kids that day.  I would recommend using this SAT study site: http://www.majortests.com/sat/wordlist-01.
  4. Lastly make some cards or stickers with your set of words and celebrate your own Penultimate Day of School.

That’s it.  Nothing complex, just a little fun for the next to last day of school. 

In the spirit of fun I’ll end with this quote I found that uses the word penultimate along with one of my perennially favorite comics Peanuts, “Although Brown intellectually knew that Lucy had demonstrated a repeated propensity to whisk away the football at the penultimate moment, leaving him tumbled head over heels, he was ever hopeful that she would hold true the very next time.”

Neil Trautwein Feb. 2, 2010

Charles Shulz

Happy penultimate day of school!

Related Links

Nothing to do with S+S – I Love the Word Penultimate (2009)

]]> Leaving Office, Joining Bing, and Planting Asparagus https://docs.microsoft.com/archive/blogs/kenj/leaving-office-joining-bing-and-planting-asparagus Sat, 27 Mar 2010 10:43:13 GMT https://blogs.msdn.microsoft.com/kenj/2010/03/27/leaving-office-joining-bing-and-planting-asparagus/ image Sometimes it’s good to get outside, get your hands in the dirt, and plant things, especially if you can do it with your children. Keeps you grounded, and yes that was an intentional pun.

 

Office 2010 is going to be great. For the past 3.5 years we’ve been working on an amazing set of features like Office Backstage, SharePoint Workspaces, the Office WebApps on SkyDrive, and even slipping in late breaking new features such as the Outlook Social Connector.

image

My first official day with the Office team was Nov. 1st 2006 and the ship party was later that same week. The weather wasn’t cooperating but that didn’t dampen the spirit of the revelers. Most of the action was happening in the open area between buildings 16 and 17 which some have dubbed the walk of fame because of the tiles embedded in the ground denoting all the products shipped by the Office organization. Walking around at that time I saw the joy and excitement of the Office 2007 team but that wasn’t my release, I was a newbie coming to hang out, I was signing up for the next wave, wave 14.

image

 

The next logical number for Office would have been 13 but I’m guessing a few managers suffered from triskaidekaphobia and so we called the project O14. It was meant to be a targeted two milestone 2.5 year release. Well I can’t really share all the internal tribulations that added a year to our plans but we’ll simply say much of the disruption came from external European pressures.

 

Triskaidekaphobia what?

During the development of Office 14 (that was our internal name for the project) I have had the great fortune of managing the Office Internet Platforms and Operations Team. The tool we developed to help with server diagnostics, the Unified Logging Service (ULS), is now the most popular logging tool for services with teams from OfficeLive, MS Online and potentially in the near future XBOX Live implementing it in their live services. Our work to improve enterprise monitoring with deep integration to Microsoft System Center Operations Manager has yielded significant positive feedback from early TAP partners and general surprise that we will be shipping our MOM Packs with the server products and in multiple languages. These items tend to ship months after the product. Thanks to Eric Wong for leading the dev team as we took on this project years ago.

Office service operations was the other major aspect to my job in office. I managed a great and dynamic operations team that supported services from CRM Online, to Dynamics Online, Office Live, the WebApps I already mentioned and http://Office.com. It is truly enjoyable to be involved in service operations because you get to see the end result of thousands of good and bad decisions. From the Ops perspective you suffer under all the bad decisions and are held to account for every production hiccup along the way.

 

Ops beware; unpleasant matter tends to roll downhill

and you are near the bottom

.

Fortunately I come from a testing background and am already jaded by the blame game. My attitude about a bug that is missed by testing is that if the developer didn’t write it in the first place or if the PM hadn’t created a flawed design I wouldn’t have had to find all the other bugs that kept me from finding this one that everyone seems to be quite up in arms about today. Operations tends to feel pretty much the same as test except they are further downstream with more unpleasant matter tends to float past them than any other discipline. About two years ago I created the below joke to help my operations team know that I understood their pain.

image

All kidding aside, developing and releasing a service on time and delighting customers is a joint proposition for all engineering disciplines. We can feel dumped on at times however we are all working toward the common goal of delighting customers and building successful businesses.

Given my role I spent a great deal of time connecting with and learning about service oriented architecture and services testing with colleagues from other teams. This wasn’t my first time working on server and services but I realized the world around me was pulling ahead and I needed to keep up. That drove me to dig deeper into the world of online services and cloud computing and develop strong professional connections to many individuals in the Bing and Azure product teams.

Office is laying down the plans for O15 and there are even more service experiences coming along. I am truly excited about the upcoming Office 2010 release and excited to start afresh on the next release. Over the past several years I was able to move from being that individual at the ship party that was more of an observer to becoming a real member of the Office management team.

When it came time to announce I was moving on my friends on twitter and linked in found out before most folks in Office got the news. 

That was why it was hard to choose to move on from Office but exciting to join Bing. I quite simply love services and well, Bing is pushing the envelope on all sorts of service design. That and I just can’t wait to turn the tide on GOOG.

So what’s up for me in Bing?

I’m a plumbing kind of engineer. One of my first services was the MSN Billing system and after it launched I really wanted to show my folks what I’d been working on so I got a copy of the past due account email template and forwarded it on to my mom. I was able to tell her that if she stopped paying her MSN dial up bill she’d get an email like this and it would be my service doing it to her J. Needless to say she was proud of her son if not a bit confused. Once you’ve gotten over the need to have the world see you elegant design it become easy to work on plumbing.

Testing Bing plumbing is what I’ll be up to. I’m not yet clear on how much we Bingers (I’ll have to check on what we call ourselves) share publicly before I go into any more details. In the short run though I’m back into learning mode reading The Search by John Battele (almost done), Agile Testing by Lisa Crispin and Janet Gregory, and re-reading The First 90 days by Michael Watkins. I discovered a cool feature on LinkedIn to share your reading list so all the books are posted to my page there.

While I’ll miss the Office team I am looking forward to our Office 2010 ship party in a few weeks and equally excited to be setting of on another great adventure.

I’ll be presenting at the Better Software and Agile Development Conference this June 10th and may have a few new stories to weave into my talk on Testing in Production and the demise of the traditional test lab. Stay tuned for more posts as I find time, now I have to go plant asparagus with the kids.

]]> Telemetry Myopia: The Toyota “Fail Safe” system Debacle and Google cavalier attitude toward Privacy https://docs.microsoft.com/archive/blogs/kenj/telemetry-myopia-the-toyota-fail-safe-system-debacle-and-google-cavalier-attitude-toward-privacy Sun, 07 Mar 2010 12:30:00 GMT https://blogs.msdn.microsoft.com/kenj/2010/03/07/telemetry-myopia-the-toyota-fail-safe-system-debacle-and-google-cavalier-attitude-toward-privacy/ For the past few weeks members of my team and I have been meeting with key individuals in Bing, Microsoft Online services, and Windows.  In those meetings we have focused on rich telemetry.  We’ve talked about technologies to collect data, privacy concerns and how to architect to support user privacy, and the data platform needed to process terabytes of data per day. 

One of the most well known of our telemetry systems we call Watson but is known to end users and OEMs and Windows Error Reporting.  This telemetry system sends us post crash debug information when something fails in Windows.  It is a key source of data for us to identify compatibility issues with applications and drivers as well as needed bug fixes in Windows and all our applications.  In Bing we have all the ad click data and other rich logging on par with Google.  That data is vital to running and improving Bing in this hyper-competitive market of search however we also feel strongly about end user privacy and have put in place what we feel are much stronger privacy policies than our competition.  On this point just ask Asa Dotzler, Mozilla’s Director of community development about his reaction to a recorded statement made by Google CEO Eric Schmidt, ""If you have something that you don't want anyone to know, maybe you shouldn't be doing it in the first place... If you really need that kind of privacy, the reality is that search engines, including Google, do retain this information for some time."  See the video here and read a good article here, “Mozilla exec suggests Firefox users move to Bing, cites Google privacy stance. Deal with Google provides most of Mozilla's revenues”

The point of this blog however is not to explain completely about the importance of telemetry in services or to dig into privacy policies.  I saw an article a few weeks back that made me think about the risks of over-reliance on telemetry.  In February ABC news ran a story about the results of some tests by David Gilbert, an automotive technology professor at Southern Illinois University.  In this story, the professor showed not only how could the Toyota cars experience sudden acceleration but that the problem might not be logged into the car’s black box system and fail to trigger safety mechanisms.

Gilbert says the flaw "prevents the car's onboard computer from detecting and stopping certain short circuits that can trigger sudden speed surges." ABC News adds, "he can recreate a short circuit in the electronic throttles of Toyotas that can create a surge of acceleration but can't be detected by Toyota's electronic sensors."

ABC News: Expert says electronic design flaw to blame in runaway Toyota models

February 22, 2010

In another article by the Associated Press that commented on Toyota’s secretive nature about their Black Box data the authors wrote.

SOUTHLAKE, Texas - Toyota has for years blocked access to data stored in devices similar to airline “black boxes" that could explain crashes blamed on sudden unintended acceleration, according to an Associated Press review of lawsuits nationwide and interviews with auto crash experts.

The AP investigation found that Toyota has been inconsistent - and sometimes even contradictory - in revealing exactly what the devices record and don't record, including critical data about whether the brake or accelerator pedals were depressed at the time of a crash

Toyota secretive about 'black box' data

Curt Anderson and Danny Robbins Associated Press Writers; March 7, 2010

And to be fair, Toyota has come out with a statement rebutting the findings of Professor Gilbert that you can read here.  In short, they say his re-wiring to create an environment with sudden acceleration that wasn’t logged in their systems was not one that could occur through regular use of a car.  In other words his results were flawed.

My take away from the Toyota mess and the “Fail Safe” system is to be careful not to completely rely upon telemetry.  Don’t get me wrong, I see rich telemetry data as changing the way we develop, test and ship both software and services.  It allows us to be leaner in our processes and further enhances our ability to test in production.  The thing is, it is just a data stream. 

When you take your eye off the customer, when you rely too much on data and don’t hear the complaints or concerns of end users you can end up making foolish statements about privacy such as the Google CEO or you can become secretive with your data and insular like Toyota.  The measure of the success of a product is always in the hands of the customer making the purchase.  Telemetry is great as long as it does not overcome our other senses making us myopic. 

That’s it.  I just needed to document this thought and the links.  When I’m done with my research on Microsoft telemetry systems and we design what we are planning for Office 15, I’ll try to post again on this topic with design recommendations and personal lessons learned.

]]> Testing in Production – It’s a living ecosystem, not a patient on a gurney https://docs.microsoft.com/archive/blogs/kenj/testing-in-production-its-a-living-ecosystem-not-a-patient-on-a-gurney Wed, 10 Feb 2010 23:41:00 GMT https://blogs.msdn.microsoft.com/kenj/2010/02/10/testing-in-production-its-a-living-ecosystem-not-a-patient-on-a-gurney/ James A. Whittaker’s (JW) recent blog post “Testing in the Data Center (Manufacturing No More),” is a fun little ditty and it inspired me. Thank you James for sharing the testing is like an ER analogy.

In this post JW draws an analogy that software is moving from a manufacturing paradigm to one of a health model paradigm. Though I’m going to pick at a few pieces where the analogy falls short I have to admit up front it’s a great juxtaposition. All of his talk about the cancer of security, products that die of natural causes in the datacenter and OH the clip board of data are truly awesome. Really, I recommend reading the his blog post here and then continue to Bing for better results and a cooler UI with neat daily pictures. Sorry, but I tend to be a bit partisan.

UG09TheWorstCaseScenarioSurvivalCardGI’m a big proponent of blurring the line between pre-release testing and testing in production (TiP). I’m not a fan of Big up Front Test (BuFT) that my friend Seth Eliot posted on a few weeks ago. I do like what JW did with the whole ER and tesitng thing but it didn’t quite hold together for me.  I think I need to find my own crazy metaphor for services testing.  Perhaps, testing in the wild Amazon basin. Nope can’t use that, another competitor.  Perhaps I should write a post on how to survive the worst case scenario while testing in the Wild Australian Outback. Nope, I’m a bit more pedantic with a bit less flare than JW.  I do however like this natural environment and survival angle.

My current focus within Microsoft services testing is all around improving release velocity while mitigating risk. Yes, a bit pedantic compared to the ER analogy but go with me for a bit here.

BTW, did you know there is a bit of a revolving door between Microsoft and GOOG? I have quite a few former colleagues around various companies. Some years back, I remember talking to Pat Copeland, who is currently in a very senior management position within GOOG, about the role of test as the driver of efficiency and arbiter of risk mitigation as opposed to the quality control expert. I argued that test had the data and so we had an obligation to play in the efficiency space and inform management of risk. As I remember the discussion, Pat didn’t agree with me that test was responsible for efficiency but he didn’t fully dismiss the risk piece.

I really like JW’s analogy to a hospital wing with patience in beds and whole wards of the hospital dedicated to really big products. It is all very visual.  Where I find the analogy falls short is that JW seems to treats a service as a single patient in the hospital dealing with their set of symptoms. The challenge we have in the services world in interconnectedness. There is no such thing as a standalone service anymore. Put simply, “No man is an island.”

No man is an island, entire of itself; every man is a piece of the continent, a part of the main; if a clod be washed away by the sea, Europe is the less, as well as if a promontory were... Any man's death diminishes me, because I am involved in mankind; and therefore never seek to know for whom the bell tolls; it tolls for thee." John Donne, Mediation #17, 1623.

Every online service is becoming interconnected. Even applications such as those on various app-stores or even Microsoft Office (I’m close to shipping Office 14) on the PC are becoming reliant upon services for the complete user experience. Neither Hotmail nor Gmail can function successfully and cost effectively without a host of other services such as federated authentication, advertising, and personalization as integrated pieces of the total experience.butterfly_effect

No service is an island, or a single patient lying on a gurney. They are interconnected. As certainly as the beating wings of a butterfly (see butterfly effect) can cause a tornado halfway around the world, each and every service can have an impact on the next.

So, if not the singular patient metaphor, then what?  Let’s go environmental, shall we?

I think perhaps we should apply an environmental analogy and given my encounter with Pat Copeland all those years ago I might change risk mitigation to environmental protectionism.

We often have clusters of services that are more interconnected than others.  We could consider these clusters to be like little ecosystems. the are heavily dependent upon each other but they are not isolated from the impact of other services.  I love to point out how Mashups build upon services in ways the developers never intended.  Work by one group is consumed and distributed by another entity.  No, this is not quite the circle of life but I am talking about layers and interconnectedness.

We know for certain that even the tide pools that surface at low tide do not function in complete isolation. Each tide cycle some fish come and go, sand washes in and out, and rather importantly the oxygen within the tide pool is replenished.  The tide pool ecosystem therefore cannot exist without some level of interaction with other ecosystems.  The beach grasslands may house predators and the coral reefs just a bit further out to sea may shelter fish when they are not in the pools.  No service is an island unto itself.

Each summer my family and I go to throcke Oregon Coast and stay near Haystack Rock in Cannon Beach. Each year my wife, my kids and I visit the tide pools at Haystack Rock. Every year we stop off and visit the folks in the pickup truck that sets up the telescope and brings the plastic bins with some of the sea life in them so we can see them up close and touch a few. There is always a jar out for donations to preserve the Haystack Rock natural preserve and every year we take what cash we have and let the kids make a donation.

SeaAnemoneIt is an awesome vacation, but I will tell you that every year I go there, I do notice changes. This year the amount of seaweed was larger than any previous year i could remember.  The tide pools, though a strong and vibrant ecosystem, are ever changing and evolving. More sand comes in and a pool completely fills, where there used to be a dozen starfish now there is a solid wall of them hanging onto the underside of a large rock, and sometimes where we had seen a floor of sea anemone we now see an barren hole filled with salt water.

Software as a Service is alive and ever changing. It requires an amount of protection and stewardship or it will decay and fail. It requires vigilance or one element of the system such as sea lions may find an easy meal by hunting wild salmon trapped up against the door of lock.  the sea lions will let their ravenous gluttony get carried away to the point that there are not enough salmon left in the run to sustain them.  Similarly one service may consume too many resources of another service and cause it irreparable harm.

In my world we still have clip boards filled with data but they are ones that have the our record counts, our observations from the field, and our assessments of risk both for my tide pools at Haystack Rock as well as the risk to our service within itself and potential risk due to encroachment or failures of other interconnected systems.

Yes, I’m confident in test we are still have a major role to play in the risk assessment business, just that in the world of services, with fast paced changes, we also need to show up every day and evaluate the real world, the data center and the software within, for its health.  We need to run our tests against production, check our PH balances, count our fish, and assess our production telemetry.  At the end of the day we file our report and get ready to do it again tomorrow because the ecosystem never stops changing. 

Thanks JW.

]]> Confidence is High and Feeling Feisty https://docs.microsoft.com/archive/blogs/kenj/confidence-is-high-and-feeling-feisty Wed, 13 Jan 2010 22:18:00 GMT https://blogs.msdn.microsoft.com/kenj/2010/01/13/confidence-is-high-and-feeling-feisty/ This is a bit of personal musing.  Feel free to skip.

Originally I was going to tweet this but then I started coming up with quite a few tweets and decided a short blog would be better.  There is such a thing among testers on twitter as tweeting your own horn a bit too much.

I have a test talk to give tomorrow.  It’s an internal Microsoft talk to a bunch of senior Test Manager and Architects.  It’s for this group called QTEC that we started about two or so years ago with the help of Grant George, Alan Page, JW, and a few other folks.  It’s a cool group with some really great talks.  It will be my first time presenting to the group but as I know most of the members quite well I feel like it should be familiar turf. 

The talk is to be me presenting how to test services or is it S+S or Client and Cloud.  My thinking on the subject has evolved quite a bit since I wrote on the topic in, "How We Test Software at Microsoft."  but that thinkin has evloved because of my interactions with other great, smart testers.  @Setheliot is top notch on experimentation.  @venkatna has taught me quite a bit in his talks about how they tested Bing.  I'd mention more but they don't tweet or blog that I'm aware of.  My conern is that I have just picked others brains and so now I get to stand in front of a crowd and pontificate (I love that word almost as much a penultimate).  There must be more and I think that’s why I’m feeling feisty.

 

I wouldn’t be a good tester if I didn’t have opinions and if I didn’t then test those opinions out.  I mean I don’t want to get as feisty as @Jamesmarcusbach and try to take on anyone that has a differing opinion on exploratory testing (apparently now referred to as ET but not the type that goes home).  I don’t want to get into the kind of back and forth as chronicled by Seth Eliot in his great blog post, “Your Software Has Bugs,” in which he chronicles the debate on Code Coverage by a few more public testing luminaries. 

I’m feeling feisty though.

I do belive that I will spice things up a bit.  I really do strongly believe certain things about services testing and Microsoft’s approach to it.  I believe we have a great chance to beat our major competitors and that test can be a key asset in helping us move our company forward.  I believe quite a few things about services testing.

1.       The traditional on premises test lab is dead and we should move all equipment to data centers and VMs

2.       All load testing should be done in production and if you’re a v1 service just order production hardware or VMs 6 months early and conduct your load tests on that hardware.  Buying test equipment for load testing is a waste.

3.       Ship into production more often and finish your testing there.  Test lab results are good for unit and basic functional testing but integration testing needs to happen in production.

4.       The OneBox is the key to agility for services testing

5.       Emulation is the second key to agility for services testing

6.       Test automation needs to run against production

7.       Deployment break = build break

If you want a regular dose of feisty check out I. M. Writes blog

 

Yep, I’m feeling feisty.

 I think that’s got to be the why me part.  I’ve got red hair and at times a hot temper.  I have opinions and I don’t mind ruffling a few feathers when the mood strikes.  I don’t quite know what I’m going to say tomorrow.  I’ve got some time to work on my slides.  If Alan Page reads this post before tomorrow he’s going to be nervous.

Related Blog posts

·         TIP-ING SERVICES TESTING BLOG #1: THE EXECUTIVE SUMMARY

·         1+1 Redundancy Just Isn't Good Enough

·         The Demise of the Test Lab

]]> Stupid Service Oriented Architecture (SSOA) #1 – The 50/50 upgrade https://docs.microsoft.com/archive/blogs/kenj/stupid-service-oriented-architecture-ssoa-1-the-5050-upgrade Sat, 26 Dec 2009 12:17:00 GMT https://blogs.msdn.microsoft.com/kenj/2009/12/26/stupid-service-oriented-architecture-ssoa-1-the-5050-upgrade/ In the web services world I’ve heard a lot of really stupid things said by a lot of really smart people.  I will admit, without details, that I have been guilty of saying some of those stupid things myself.  I like to believe that I am learning and evolving though as I now recognizes those previous statements as being less than wise.

One area ripe with absurdity is web service’s deployment.  I could go on and on for pages on the number of foolhardy things I have heard, seen, and done with respect to service’s deployment.  For today, however, I choose to pick on just one, the 50/50 upgrade.

 I have heard this one time and again, and even last week.  It is often said with perfunctory conviction, a puffed up chest and a broad beaming confident smile by the engineers that have finally implemented it.  That statement goes like this, “We finally have a great deployment solution, we can do 50/50 upgrades across two datacenters.” 

Yuuuck!!  Every time I hear this, I wrench handfuls of hair out of the pitiful amount remaining about my head, and run screaming from the conference room.  It is perhaps the most pathetic-neanderthalic-misanthropic  statement an engineer could make.  What really gets me is how it is always said with such bravado, such confidence, such unthinking belief that this is a great idea, that you suddenly realize the engineer saying it and those nodding their head like little Chihuahua bobble heads, really believe it.    

We have a great deployment solutions

we do 50/50 upgrades

 

So what is 50/50 upgrade you ask?  It works essentially like this.

A web service is built out with all the components needed to function and often this includes a set of web front end servers, maybe a middle tier server and a database of some sort on the very back end. Separating out component of a web service is a common architecture both in the enterprise, traditional online services and even within cloud computing.  It allows for improved utilization of hardware resources, improved security by cordoning off critically sensitive customer data, and when done correctly allows for more rapid development on each sub-component.  Even the Microsoft Azure service works on this model. 

Azure  

A v1 “new” service may make the mistake of speed to market over maintenance features and thus launch with an architecture that is tightly coupled and in massive flux.  This will force the development team into many rapid releases that cause the entire stack to change.  Applying upgrades to every server in production flows from the service pack mindset of the enterprise where a poor operations engineer has been assigned the task of apply the new service pack or security patch to every machine in production by the end of the weekend.  Apply the upgrade everywhere, innovate everywhere in tandem, upgrade to the new version of the service all at once so we know where we are...

By the way, I really can’t stand tight coupling and all or nothing architectures.  Perhaps I’ll get to another post on general SOA concepts as opposed to this one about SSOA. 

This is a bad approach for web services and leads toward tight coupling and monolithic upgrades.  Monolithic upgrades of a service often result in the well know, “Sorry, our website down for maintenance.” sign being put up for the world to see.  These kinds of signs range from somewhat cutsie to downright ugly.    

maintenance 1Maintenance 2

The maintenance window is actually an acceptable option.  Many major web services have used this concept.  Everyone from parts of MSN, eBay, and even SalesForce.com has used the maintenance window concept of a weekly or monthly basis.  The problem is, it is a crutch and the web service really needs to focus on how to have zero downtime deployments.

The decision making process on how to move to a 50/50 upgrade model tends to go something like this:

1.       We can’t keep taking these maintenance downtimes.  Our competition isn’t down, why are we?  Get me one of those zero downtime deployment solutions and get it now!

2.       Oh, and we should also become geo-redundant.  I hear that is a great think in case we get hit by one of those earthquake things.

3.       Let’s build out a duplicate in a new data center so we can do a 50/50 upgrade!  All we need is an expensive network load balancer at the edges and a bunch of virtual IPs.

4.       Great, but we’ll need to build it out with enough capacity to handle all the user load

5.       That’s right and our peak loads usually come right around the holidays, back to school, or yearend accounting, or federal tax day, or Valentine’s Day.  This last example probably only affects http://hallmark.com (see Hallmark case study slides among these TiP slides here), but you get the point that most services have a peak utilization that is calendar or event driven.

6.       This will be great because if the upgrade doesn’t work well we can always fail back to the half that hasn’t been upgraded.

7.       We’ll need to build out with some buffer too because we don’t want to have a service performance slow down because we have a really big spike.  I recommend 125% of peak.

8.       Perfect, we don’t want any risk with this new 50/50 upgrade architecture.

9.       Great let’s build out another duplicate cluster.

At this point the meeting ends and everyone goes their separate ways feeling quite pious about the great decisions they just made.  The move to a 50/50 upgrade is underway.

Let me briefly explain some of the mechanics of a 50/50 upgrade.  The way it is accomplished is that a service will build out a duplicate infrastructure, often in a different data center but it could be in the same data center, but it will involve new hardware.  This mistake can be made within cloud computing by creating duplicate VMs as well.  This new build out will need to be at scale as the maintenance window might become twenty-four or forty-eight hours.  Each of these units is often called a cluster or a scale unit.  Fortunately with modern network load balancers,  such as those build by F5 Networks, it is easy to fail over from one cluster to the other, thus allowing you to take one cluster out of rotation for maintenance.  Upon completion of maintenance the upgraded cluster is put into production while the other half is taken off-line for the upgrade. 

With teams I’ve run we typically put the upgraded servers back into production overnight to see how they perform.  We want ensure the users are able to connect and use the service as expected. We also watch to see if the servers suddenly start to throw lots of errors or crash. 

Once we are sure the new service is running as expected we can begin to upgrade the remaining machines.  This gives us our rapid roll-back plan in case the new version has a major flaw and cannot be released.  50/50 upgrades as a tool for rapid rollback is the first stupidity, but not the most significant, I want to call out. 

We have great roll back plan if the deployment fails

we just fall back to the non-upgraded servers.

 

RANT: If I design an upgrade that is so invasive it takes by 25 hours to upgrade a cluster of say 100 servers, I then put it into production and it fails, and so my roll back plan is to fail back over to the non-upgraded machines, then what the heck am I to do with the mess I created on the first half?  Does anyone have an automated tool for me to get them back to the previous version, nooo.  Does anyone have a plan on how to fix the bug in the new version so I can move safely and easily forward, noooooo.  All I have is a net to land in when the service falls over, but unfortunately there is no way out of this net.  I’m stuck halfway between flying seamlessly through the air with the greatest of ease, and smashing my server guts all over the concrete floor.  If a trapeze artist falls every other time thy try to jump from one bar to the other, it isn’t much of a show.  Proper practice to get the transfer right is the way to go, the net is there for only the worst of incidents.  Teams that fall into the 50/50 upgrade trap and point to their rollback plan as failing back to the un-touched servers, are cutting corners and hoping the net will never snap. 

In the words of one of my favorite news commentators from ABC New 20/20 John Stossel, “Give me a break.”  John has a new blog here.

So, that’s the first stupidity.  The second stupidity is the one that really chaps my hide and that’s the one around frivolously adding massive cost to an online service.  In fact because of this one factor, I would rather a service stick with maintenance windows than move to the 50/50 upgrade option.  Table 1 shows just how expensive a 50/50 upgrade can be and how quickly the cost drops if you simply move to a smaller scale for upgrades such as a ¼ upgrade.

Upgrade %

# of units to upgrade

Minimum capacity for peak load

% capacity for entire service

% wasted capacity

50/50

2

100% + 25% or more buffer

~250%

150%

1/3, 1/3, 1/3

3

50% + 12.5% buffer

~182.5%

82.5%

1/4, 1/4, 1/4, 1/4

4

33.33% +8.33% buffer

~166.5%

66.5%

Table 1: % of wasted capacity moving from a 50/50 upgrade to a minimum of four clusters with shared redundancy.

With a typical 25% capacity buffer a 50/50 upgrade scenario will push a team to build out a minimum of 150% extra capacity.  Remember that this is 150% wasted capacity above peak utilization that might come just once a year.  For Microsoft Office we know our biggest peak is right around back to school in the early fall and so we have extra capacity available at http://www.office.com for this huge spike but don’t do 50/50 upgrades.  The real waste in terms of such capital is probably great that 200% and then add in the operational expense of powering and maintaining that much extra capacity, a service can easily go from being cost effective to cost prohibitive. 

This is the stupidity I really loath as it is so obvious and yet never really considered up front.  For a little extra effort a team can often engineer their deployment scenario to the point they can move from a maintenance window model to a N cluster deployment option where N is great than four.  That is what I point out to ever team I work with that falls for the 50/50 trap.  Still many ignore me and take the easier route of throwing hardware at the problem.  The problem is that this bad decision when made, will sink capital and actually limits the team’s ability to be more innovative through lose coupling and more dynamic and automate deployments. 

I’m planning or writing a few more blogs on Stupid SOA and other Stupid things about services.  If you have some to share, I’d love to hear yours.  I’ve also tweeted some gems on my twitter account at https://twitter.com/rkjohnston. Tweet, comment, send me email, but whatever you do, if you have a good one, please send it to me. 

Ralated Blog posts

]]> Bing Outage exposes Testing in Production (TiP) Hazards https://docs.microsoft.com/archive/blogs/kenj/bing-outage-exposes-testing-in-production-tip-hazards Fri, 04 Dec 2009 10:28:00 GMT https://blogs.msdn.microsoft.com/kenj/2009/12/04/bing-outage-exposes-testing-in-production-tip-hazards/ I have been a big proponent of shipping services into production more frequently and conducting more testing on the code once it is in production.  Those of us that support this methodology tend to call it TiP (Testing in Production).  Find links to previous blog posts on this subject below. 

Bing During OutageBing back up and working

After the recent Bing outage (evening of 12/3/2009), I find myself thinking about the Hazards of TiP and thought I might make a post about some lesson's I have drawn from this production outage and what has been written about it so far. ZD Net posted a bit of a sarcastic blog with the title "Microsoft is making progress on search: You noticed Bing's glitch."  According to the official blog post by the Bing team (here) the outage was, “The cause of the outage was a configuration change during some internal testing that had unfortunate and unintended consequences.”

Internal testing that had unfortunate

and unintended consequences

 

Despite this black mark, I still believe TiP is the right direction to go for services testing but clearly there are some hazards and lessons we can extrapolate.

These two posts imply that the outage was wide spread, noticed by a lot of individuals, and caused by an errant configuration change in support of a test.  My assessment is that while there was clearly an attempt to run a test configuration in production, the test did not cause the outage.  The challenge came where the test configuration change somehow went to all of production. 

The core concept of TiP is to minimize risk through TiP-ing.  In order to accept the risk of less stable code into production in order to run tests, the less stable code must be easily sandboxed.  Whatever happened here was likely a configuration management mistake not a testing error.

I have an axiom for my operations team, and that is that all manual processes will eventually fail, unless they don’t.

I like Bing, but half-an-hour downtime is unacceptable these days. Do you guys not have failover systems?

Comment on Bing Blog by b.dylan.walker

 

The reality is that the Bing system is very automated.  The team has shared some information about their infrastructure so I won’t go into details here less I share something not disclosed.  In outage like this from a test configuration change impacting production is clearly a case of fast moving automation. 

In order to enable TiP and to take more risk into production, the change management system of a service must be rock solid and fully automated.  Clearly though from what has been shared they have a state of the art system.  In fact it is likely this state of the art system that allowed the errant change to propagate so quickly and require a full roll back.

Therefore the gap must be in the safety mechanisms to prevent such a mistake in combination with how fast the mistake rolled out to all environments.  Another factor in successful TiP is metering of change in production.  This change just moved too fast and while the bing system is highly automated it still takes a long time to undo a change across so many servers.

My takeaway from this outage is to remember that TiP does work but you need solid change management.

1.       A fully automated deployment system

2.       Rock solid controls on change management approval

3.       Every change must be a metered change so when a mistake does happen it doesn’t affect every server in production.

 

Those are the lesson’s I’ve drawn so far.  What do you think?

Other Blogs on TiP

·         TIP-ING SERVICES TESTING BLOG #1: THE EXECUTIVE SUMMARY

·         Ship your service into production and then start testing!

Images from a blog post on WhatWillWeUse.com - http://whatwillweuse.com/2009/12/03/hold-on-40-minutes-while-i-bing-that/

]]> TIP-ING SERVICES TESTING BLOG #1: THE EXECUTIVE SUMMARY https://docs.microsoft.com/archive/blogs/kenj/tip-ing-services-testing-blog-1-the-executive-summary Tue, 24 Nov 2009 14:28:00 GMT https://blogs.msdn.microsoft.com/kenj/2009/11/24/tip-ing-services-testing-blog-1-the-executive-summary/ For some time now I’ve been promising to lay out all my thinking on Testing in Production (TiP).  I first introduced the topic in a blog post in June 2009 titled “Ship your service into production and then start testing!” Over the summer I had some family commitments, late summer I finished up my internal Microsoft ThinkWeek (full article in Seattle Times here) white paper on the subject (5 five star reviews so far), and had to prepare a couple of talks on the subject for STARWest and the TwinSPIN & Benchmark QA SIG in Minneapolis.  All those excuses are over and now I’m ready to get back to the blog. 

 

 

What follows is an updated version of the executive summary of the ThinkWeek paper that I co-authored with Seth Eliot (Seth's new blog can be found here) and Ravi Vedula on TiP-ing Service Testing.  The paper did quite well as we recieved numerous 5 star reviews and the chair gave us a four start review with a must read comment.  The paper has seen over five hundred views so far, so I'd call it a success.  Ithant is is my best overview of the subject and makes the case for why TiP is an important framing concept for evolving our approach to services testing.  For more on ThinkWeek check out the youtube video or news article links above.

 

“How could Test have

missed this bug?”

 

When it comes to services testing there are two diametrically opposed perspectives.  The most common perspective is the one that is focused on making test environments in the lab as close to production as possible.  This approach grows out of the oft expletive laden post RTW (Release to Web) admonishment, “how could test have missed this bug,” and the equally common retort, “our test environment isn’t enough like production so we couldn’t catch it.” 

 

Many a tester has been burned by the bug that was missed and when this happens they often become defensive and more risk averse.  They tend to drift into the first school of thought that attempts to continually increase the precision with which their test environment matches production.  All sorts of techniques are used from purchasing the exact same servers for testing as production, purchasing load balancers and even taking sanitized dumps of production data into the test lab.  After all, it is the job of testing to make certain bugs don’t get away and if even a single bug is missed in test because there was a missing network device or we only had one SQL Database instead of mirrored databases like production, then we must close that gap and stop the bugs from getting out into the wild.

 

“Test is the gate keeper, the champion of the customer, 

and must stop all bad code from escaping!”

 

The funny thing is, whether it is a game, a desktop application or a web service, if you are a tester that has shipped a product, you have missed a bug.  I know I have missed a lot of bugs in my career, and yet I remain employed.  While I too work to find all the bugs in a product, when I hear a manager ask that same tired question, “How could test miss this bug,” I tend to take a deep breath and just let it pass. 

 

Personally I am always confident that my team and I have done a good job and so my advice to others is to just let the comment pass.  Think of it this way, it’s not your fault.  If the bug hadn’t been designed in the first place or coded in the second place it wouldn’t exist.  The fact that it exists is not your fault.  You found tons of bugs in the product, so a few got past, go yell at the development team for daring to write the bug in the first place.  When you become defensive you become obsessed with never missing a bug.  I have a long talk and even a tutorial on what we can learn from bugs that get away, for now just let it go.

 

 TiP Challenge

 

The second perspective is the one that accepts test can never find every bug in the test lab and with respect to services; test can never fully emulate production.  Trying to find all possible production bugs in test is a daunting, expensive and eventually un-achievable aim.  There are simply a vast number of bugs that will only ever be found in production.  This line of thinking pushes one toward considering risk mitigation once a bug is out.  After accepting that bugs will get out you can free your mind to start thinking about how to leverage production for testing and how to make production more test friendly. 

 

TiP is really a paradigm shift requiring the acceptance that test environments have limitations, test environments cannot catch all the bugs, and testing in production is the most viable option.

 

I should probably pause a moment and define what I mean by Production and Testing in Production. 

 

Definition of PRODUCTION: For this paper production will be the current version (v-current) of the service to include all the data centers and machines v-current runs across.  It will also include v-Next instances that live in the data center and have real world traffic hitting them.

 

Broad Definition of TiP: TiP constitutes all testing activities occurring on hardware in a data center. 

 

To most testers and operations engineers the notion that you would test in production seems verboten.  TiP is an anathema.  TiP is just not allowed.  The reality is that it is allowed and it is much more common place than many think.  In fact this paper will argue that we need to further invest in and enhance our abilities to test in production. 

WLID Test Account

Figure 2-Test Account used frequently on Hotmail for production testing

 

If you were to look at TiP along a continuum at one end you would find the “Lab Centric” perspective and at the other you would find the “TiP Zealot.”  The Lab argument would focus on making the lab as much like the production as possible.  If we are looking at Windows or Office that would include all kinds of hardware and peripheral devices, all kinds of applications for compatibility testing, and thousands of machines running automated tests.  To be fair, this has worked well for Microsoft with respect to desktop applications and server products.  Even so we work with beta partners for them to deploy the next version of our products into production so we can get early feedback.  In this case though, production is often treated as a backup to testing in the lab; the test focus is still within the test lab. 

 

This approach to testing has been used by Microsoft and many other companies as the primary approach to testing web services.

 

The TiP zealot would argue that the lab will never be able to replicate the real world so all final sign off must happen in the real world, in production.  Looking again at the software side of things we see the example where we run betas that sometimes include millions of customers and identifying selected Technical Adoption Program (TAP) partners to sign off on final Release to Manufacturing (RTM).  

 

TiP Continuuum

Figure 3: Services TiP Continuum show the shift to more production testing

 

The reality is whether we are talking software or a service we already conduct much of our testing in the real world and in production.  What this series of posts attempts to do, is to organize a collection of best practices within the services space that will move us further along the TiP continuum safely toward a TiP centric perspective.

 

Future posts in this series will focus on three main sub-section from the original white paper.  These upcoming sections are essentially the why, the what, and the how of TiP:

·          Why – Factors pushing us toward TiP

·          What – Best Practices for TiP

·          How – Enabling technologies for TiP

 

Why – Factors pushing us toward TiP discusses how factors such as Scale and Integration are forcing many services to test in the data center and even forcing them to test the data center itself.  Technology shifts such as Cloud Computing make it easier to test in production and other shifts such as an ever deepening stack of infrastructure layers make it impossible for us to replicate production in test labs. 

What – Best Practices for TiP is likely the most fun for readers as I will simply lay out my top TiP Tips as a collection of best practices.  These practices will cover everything from the simple idea of purchasing production hardware in the data center that is used both for testing and production interchangeably to the relatively bold concept of shipping test hooks into production.

 

How – Enabling technologies for TiP will wrap up this series by outlining some areas where we need new enabling technologies and improved architecture.  Many of these solutions should be built into the coming wave of Cloud Computing offering being made by just about every major software and Telco company in the world. 

 

As I’m building this series off of the white paper written by me (@rkjohnston), Seth Eliot and Ravi Vedula the next installment should come out fairly soon.  In the meantime if you have questions about TiP or just want to add your thoughts, please post a comment.
]]>