It was 1998. I was a middle manager for a platform called CDO, Collaboration Data Objects. The API’s were to allow Application Developers to build fantastic workflow tools on top on Exchange Server. The platform was available on client and server OS’s. If you’ve ever used the mail API’s in .NET or CDONTS on NT, you’ve used it.

Exchange Server back then partnered with the Outlook and Windows Server teams. CDO was going to be newly shipping in Outlook and the (then) upcoming Windows release. My team and I were all very excited about it. We carefully considered all of the cases, wrote up beautiful plans, drew up perfect schedules, and then…

Proceeded to execute in an entirely interrupt-driven fashion. Whatever emergency Dev had, my team had. Exchange itself had heavily marketed itself as an App platform, so whatever emergency it had, my team had as well.

This was all good until I got a call from my manager. It seemed someone in the Outlook team had found a bug in CDO. Office was a couple of weeks past locking its build down for final ship. The bar had to be pretty high for any bug to get fixed. Well, this one was.

The claim was CDO was not working on Windows 98…. At All.

I guffawed indignantly at that claim. Of course, it works! My team was the best test team ever. So I went about trying to debug what might have gone wrong. Silly User!

So I did what I had done a zillion times before, except something new happened this time….

NOTHING!

I stared at the screen in disbelief. I was shocked. I tried again. Still NOTHING!

I called in my Dev middle manager peer and asked him what he thought I was doing wrong. He asked a bunch of questions that all showed nothing. So he got one of his developers to debug.

“Oh”, he said nonchalantly, “from the callstack, you are missing DCOM. See… It’s failing to load a DLL.”

So we looked to see what was wrong with DCOM… We found out that it wasn’t available on that OS. EVER!!!

My team, being server based, had all been testing on NT. Not, you see, on the Windows 98 OS that Office had just used to sign off.

FAIL.

I was dejected. I looked over all of our plans. We planned it. But we had failed to execute on covering it. Just had not yet gotten to it and had not tracked it as an exit criteria. None of my 35 testers, 20 developers and 5 PMs had tried it.

It was definitely a bad moment. Once I realized that I was going to need recall Office from shipping on time for this, I was miserable. So I began practicing my speech and walking to my boss’ office. One of my frontline managers intercepted me. I think he knew I was going to fall on my sword. If I had been my boss at that moment, I would have fired me.

“Brent, don’t do it.”

“Don’t do what?”

“Anything. Go home. Sleep on it just 1 night. If you still feel the same way, I won’t stand in your way.”

That was reasonable. So I did. Had a nightcap and tried (failed) to get a wink of sleep. When I came in, I found the same manager there… And in the same clothes… In fact, so was my whole team. They had all done an all nighter.

You see what happened was my team did not want to see me go down with the ship. They had all worked together to find a fix and completely test it end to end in 24 hrs (including finding and fixing bugs). The whole undone section of my test plan was repaired in 24 hrs.

So yes, I went to talk to my boss the next day. And yes, I got yelled at, but my team had given me a way out of the nightmare.

This is my favorite bug because of two things:

• It reinforced the importance of testing.
• It showed me why it is *always* important to invest in and be loyal to your employees.

-- Brent Jensen

Do you have a bug whose story you love to tell? Let me know!

ANSI standard 754 mandates rounding and such behavior for real finite-precision floating-point arithmetic. But there is no standard -- at least I can say for sure there was no standard in the early 1990s -- for extending those rules to complex arithmetic. So I did a fair amount of exploratory testing of MATLAB's complex arithmetic details on various operating systems, mathematical libraries, and machine architectures.

One basic mathematical rule is that a number times its complex conjugate is always real (has no imaginary part). So I wrote a little test that multiplied a bunch of numbers against their complex conjugates, and looked at the results. Nearly all platforms performed as expected. But the IBM RS-6000 would give Funny Answers: often there would be a teeny tiny little imaginary part, on the order of ten-to-the-minus-sixteen-or-so times the real part.

I showed the result to MathWorks chief scientist Cleve Moler, who speculated that it was a problem in how C's mathematical operations were compiled down to the RS-6000's instruction set. He sent the result off to the appropriate people at IBM, who quickly determined that that was true. 

On most platforms, the part of the computation to determine the result's imaginary part compiled down to the following:

1. Multiply the first number's real part by the second number's imaginary part and store the result in standard 64-bit floating-point format.
2. Multiply the second number's real part by the first number's imaginary part and store the result in another 64-bit register.
3. Add the results of the two registers and store the result back in one of them.

But on the IBM RS-6000, the operation was fancier and exposed that they were carrying internal results around in longer (I believe 80-bit) registers:

1. Multiply the first number's real part by the second number's imaginary part, to 80-bit precision, and store the result in a 64-bit register.
2. In one shot, using the "multiply with add" instruction: multiply the second number's real part by the first number's imaginary part, to 80-bit precision, and add in the result from the register (which has been rounded to 64 bits), and return the answer.

See the problem there: the second half of the operation carried more bits after the decimal place than the first half did, so when you add them together, those low-order bits would be left, which would then come out as the imaginary part of the answer.

IBM fixed this issue quickly without a fuss.

This is my favorite bug because it demonstrated to me (and to others at the MathWorks) the value of hunting for bugs, it launched me into a long period of curiosity about finite-precision arithmetic (leading eventually to an advanced degree in numerical analysis), and, in retrospect, showed how a company could fix a bug that had significant impact on scientific computation without the widespread hue and cry that resulted from the classic "Pentium Bug" that dogged Intel a few years later.

-- J. Michael Hammond

Do you have a bug whose story you love to tell? Let me know!

Every time I have to follow below steps while testing in production:

1. Edit the Project list item and add myself as a Project Member/Owner
2. Submit the status
3. Remove my name after testing

I followed the same steps for the project for which the users were unable to submit status but I didn’t get any error.

ISSUE:

Actually the issue was, during the recent deployment in production, the developer messed up the Category field because of which the category had become empty for some of the projects. I could not catch this issue even though I tested in production as every time, I edited the project list item to add myself to the Project Member/Owner, then as the category is a dropdown list it defaulted to “X”, that “X” category was assigned to that particular project, and I didn’t get any errors while submitting the status.

After a detailed analysis I found it.

-- Mohan Sagar

Do you have a bug whose story you love to tell? Let me know!

You might remember I went to CAST last year and came back way impressed; this year's CAST managed to actually surpass my expectations! And it failed to meet them as well.

Location. CAST 2011 was up in Lynnwood, WA, just up the road from Seattle. In my mind this is starting to get into the boonies, but really it's still well in civilization. Plenty of four star (according to Bing) restaurants of all makes and models (including, apparently, a preponderance of Really Good Korean), its very own conference center, and hotels across a range of price points and amenities.

The conference hotels were fine, nothing fancy, not horrid either. The biggest problem here was that plural: there were *two* conference hotels, a 20-minute walk from each other. Not a horrible distance (I walk way more than that everyday), but far enough apart that it took coordination to decide when to meet who where. Just one hotel - or multiple in closer proximity - would have been nice. (CAST 2012, please take note!)

Food. I was impressed by the conference center's food: much better than I've come to expect from conference center fare. This would be a solid "Awesome!" were it not for the distinct lack of protein in *some* of the meals. Like about half of them. While (I guess) some people get by fine eating mostly carbs, I need pretty much a 50/50 mix between carbs and protein. Sandwiches had plenty of protein, nothing else did. (CAST 2012, please take note!)

Program. The program this year was created rather autocratically, with presentations being by invitation of the program chair rather than the typical call for papers. This resulted in a bit of an uproar, so an Emerging Topics track was added: shorter, 20-minute talks (as opposed to the 60+ minutes of standard sessions), suggested by and voted on by the CAST community. The shorter length resulted in a somewhat lower barrier to entry - while the community input on the speakers may have actually resulted in a higher barrier to entry. This was a great idea I hope is continued in the future! (CAST 2012, please take note!)

Sessions. The sessions were uniformly great - which is not to say every speaker was incredibly polished, which is exactly as it should be. CAST isn't so much about "Come hear me wow you with my awesome speaker skills (that don't give you a chance to get a question in edgewise)" as "Come hear me talk about this thing, and then let's have a facilitated discussion that's as much about your thoughts as mine". So while more polish from some of the presenters might have been nice (like, the microphone's there to be spoken *into*, not to hold by your waist), the discussions more than made up for any lack of polish. And - in my ultimate test of a talk - I came away from every talk with at least one new idea or question. So awesome!

Tutorials. I attended Michael Bolton's tutorial on Test Framing, so I can't speak to the others. In a fractal of the conference, I think this tutorial both met and failed my expectations: I finally understand what Michael means when he talks about framing tests, and every encounter I have with Michael ends up making my brain hurt in a really good way; yet it also felt like a partial rehash of Rapid Software Testing, and I spent about a third of the tutorial puzzled how it had anything to do with test framing. Overall, though, a good use of my time.

Questions. The questions I most remember - and am still pondering - are:

• "How do I decide how much debugging I should do v. handing over to my dev?"
• "Why run a test I'm never going to use results from?"
• "What place do emotions have in testing?" (one of many coming out of my lightning talk on "Emotional Testing"; more details in future blog posts)

Quotes. Oh the surfeit of fantastic quotes :) My favorites:

• Pretty much everyone, at one point of another: "All models are wrong, some models are useful"
• Michael Bolton, in his keynote: "A tester's job isn't to confirm that a product works as expected, but to discover how it actually works"
• James Bach, in his keynote: "Doing something just because you're told to isn't being an engineer"
• Ajay Balamurugadas relating how a company asked Weekend Testers to test its product, they did and replied with a plethora of bug reports - and then never heard from said company again
• Pete Whalen, in his Emerging Topics talk, suggesting doing integration testing "like your system's a character in Pulp Fiction, because all those 'unrelated' systems actually are"
• Paul Holland quoting Paul Gerrard's modification of a quote from Michael Bolton: "Why didn't I find that bug? Because the developers hid it so well even *they* couldn't find it!"
• Nancy Kelln amazing list of ("truly awful", to again quote Paul Holland) testing-related metrics
• Sherry Heinze's understated "It's hard to test if you don't think"

Am I glad I went? Definitely! Even with inexperienced speakers, protein-light meals, and half of CAST being in a different hotel, this was the most fun - and learning - I've had all year. The keynotes, Emerging Topics, and (I think) Lightning Talks will soon be posted at CAST's blip.tv archive; I encourage you to check them out and get a taste of what could be yours next summer when CAST 2012 comes to Silicon Valley - hope to see you there!

The short version: CAST is awesome, conference proceedings will soon be up at http://www.cast2010.org so you can see what you missed, you should go next year!

Longer version: Monday I participated in an all-day tutorial by Fiona Charles on Speaking Truth to Power. Most of this session involved role playing scenarios about telling an unwelcome message to people in power. Some came from our real lives, others were invented; all came scarily close to real situations we had suffered through at some point.

I found this session the most helpful of any I attended. We could halt the role play whenever we desired and discuss what had happened so far, then continue on or replay it from the beginning. This let us experiment with different approaches and see how they affected a scenario's outcome. Highly valuable!

I had three big learnings that day:

• "Let me get back to you on that" is helpful in a multitude of situations.
• Our emotions will hijack us when we least expect them to.
• Changing the physical relationship between the people in a conversation will change the conversation.

Each of the next two days started off with a keynote and then continued with a full day of sessions. While one speaker turned me off when he seemed to make fun of waterfall ("Look at how long it takes!"), it turned out OK; every other session was quite good.

While the talks were interesting, the best part of every session was the facilitated conversations at their end. At least a third of every session was devoted to these discussions, and the CAST staff ensured that even the shyest introvert had a chance to talk without being steamrolled by the decidedly non-shy extroverts who always seem to dominate Q&A periods.

Extrapolating from this you might guess that pretty much everybody talked with pretty much everybody else throughout the conference, and you would pretty much be correct. Before sessions, after sessions, and into the wee hours of the night I had any number of conversations to choose from, and I took full advantage of these opportunities to meet new people, chat in person with people I had only met online, and renew friendships with people I already knew. For a person like me who loves to talk with people about testing, this was nirvana!

So, like I said earlier, CAST was awesome, conference proceedings will soon be up at http://www.cast2010.org, and you should definitely go next year! It'll be in Seattle, and I will definitely be there!

1. PM reaches the end of the milestone first;
2. Dev reaches it next;
3. And Test reaches it last

Isn't that how it always seems to be when it comes to software development?

Judging from the tweets I've seen, Twitter seems to be a combination Facebook + instant messaging tool where friending someone isn't necessary before you can see their updates; I don't see the point in that yet.

The main reason I haven't been blogging is that I'm not sure what I want to say. One thing I learned in my recent job search is that the act of testing no longer excites me - I'd much rather spend my time helping people it does excite do their job. And to answer the obvious question: I still don't want to be a manager. All of which makes my current role of Toolsmith (which inevitably involves much process-smithing) a fit while I figure out what I *really* want to be doing with my life now. (Which may turn out to be exactly what I'm already doing...)

Binging my way through the TED podcasts is helping me decide what I want to do with my life; at least, many of the talks feel important to me in a way I haven't yet put my finger on. More details when I have 'em.

As I've mentioned previously, my primary task right now is building a new automation stack for my team (~60 testers; all of Windows Phone Test eventually I'm hoping). While my thinking has evolved somewhat since I last blogged heavily about automation stack design, my first version here looked pretty similar. The more I learn about my team and their problems, however, and the more I realize that  many of the reasons I'd previously done things in particular ways no longer apply (e.g., a lack of good model-based testing tools), the more my design here changes. It's becoming simpler, mostly, which I take as a positive sign. Once my design settles down I'll post more details about the process and end result.

In the meantime, know that I'm busy writing lots of code and standards and guidelines in hopes of helping my team do their job better!

I enjoyed it! Although I guessed the major plot points and some of the details well ahead of their occurrence, I always felt like "Hah! I *knew* that was going to happen!" rather than "Well, duh - what else would have happened?" And I was surprised by other details, including the manner in which what I guessed would happen actually did.

Jerry says he switched from non-fiction to fiction in hopes of finding a wider audience for his ideas and teachings. While I don't know how he's doing reach-wise, I clearly see his teachings in the puzzles I saw everywhere as well as the situations in which he put his characters. As Jerry helps Libra and Andre deal with their various life and work problems, realize (some of) what is really going on, and discover new ways to resolve their predicaments, I see clear parallels to his non-fiction books, PSL workshop, and AYE conference. I wish he'd been writing fiction when I was a teenager! Young adults feeling like square pegs being stuffed into round holes struck home for me, and I like to think his teachings would have seeped into my brain and helped me deal with high school and college a little better! As I know is happening for me as I read his books now.

As for those puzzles I mentioned, every character and place name, as well as every sequence of numbers, had me puzzling out its significance. Sometimes this pulled me out of the story, and while I *think* I know the answer for some cases others still have me stumped. Jerry?

What I love most about the best science fiction is that it can be read on (at least) two levels: the surface story, and the underlying manifesto. Mistress of Molecules fulfills that definition of science fiction in spades. Recommended.

Over the course of these discussions, everything I heard started grouping into two buckets: Intent and Reliability:

• Intent: Our tests contain only the details that matter.
• Reliability: Test failures always mean a product bug.

Intent harks back to the stack I helped build on Expression, although I'm finding my thinking around that is evolving (more details soon!). Reliability will come in part from reorganizing our current stack and merging our multiple ways to do <whatever> into exactly one that lives in an easily discoverable location. Beyond that, we're going to put some metrics around reliability to help us identify the most problematic functions, make them reliable, then move on to the next set. I'm sure we'll find problems in our test code, in our product code, in other teams' product code, and in the underlying device code. We have good working relationships with all of these groups, and a history of working through similar problems with them, so I know we'll get everything straightened out.

In related news, yesterday I purchased a data plan for my WP7 phone, so now I can read and respond to email even when I don't have wifi. I don't think I'm turning into a phonehead though - I haven't yet found a situation where I wouldn't have rather had a tablet I could ink on rather than use a tiny keyboard that makes my fingers hurt. Heeeeeeeeeeeere, tablet tablet tablet tablet tablet! (I hope I get a shark!)

In separate related news, I'm updating my You Are Not Done Yet checklist with mobile-related items and topics. If you have any, let me know!

Recently I discovered that VPN no longer worked for me, so I dropped by Security to see what was wrong. Therein ensued a journey of frustrating details, one I am still working through.

This evening I emailed the head of Microsoft HR. The _Head_, as in she reports directly to Steve Ballmer. I detailed my experiences and grumped about them.

She emailed me right back, asking for more details. And passed my mail on to the appropriate people asking for answers.

I love this company!