For instance if I wanted to know the address of a byte[] from the following code.

You can dereference the item to get the memory address by opening the Immediate Window & entering the following:

Now you can drop the address of the memory into the memory window to see how it is laid out.

For some of technologies there may be tools you can use such as the IAccessible interface for many windows forms or the tools that we ship as part of Visual Studio Test.  Yet you may be on a product for which there are no other people testing that technology & therefore less existing items you can leverage. A tester needs to know what technologies are out there & be willing to dig in and implement similar technologies if they do not exist for their environment.

For example how can you do UI automation on an Xbox console?  The console does not have public interfaces for manipulating UI elements & is completely different than a traditional WinForm.  Additionally, it uses a different processor architecture (PowerPC) than what many Microsoft products use (x86, x64, & more recently ARM), so some tools & libraries won’t work on it.  What does it’s UI need to support for you to be able to have reliable automation?  What would you ask the developer to do for you to get better automation? Below are a couple items that you could use when designing your product to improve testability. As always with test, there is no one magic bullet but instead they are tools that we can leverage where it makes sense for our development.

Model View Controller (MVC)

Design plays a large role in how you can test a product. It influences how devs write their code & how you can then write your test code. There are many concepts out there to help improve the testability of a product design. One example is Model View Controller (MVC).  The idea with this design pattern is that you minimize the logic that is in your UI (aka View) and push your business logic into the model.  The controller then manages the interaction between the two.  With most of your business logic pushed into the model, this means you can now manipulate the model directly possibly via APIs and do a large amount of testing that way instead of relying on the UI. Traditionally API testing is much more reliable & robust than UI testing as UI changes more often and has timing issues with regards to knowing when you click on something when the next screen of the UI is ready for input.  This design also has the added benefit of allowing the developers to quickly add additional views with out much effort.

A newer pattern which is similar and targeted at several of the latest Microsoft UI development platforms (Windows Presentation Foundation & Silverlight) is Model View ViewModel (MVVM).  This adds a ViewModel which is an abstraction of the view used in data binding of the model to the view.

Mock Objects

How are you going to test error conditions in your code?  What if components you require are still under development? Are you testing a UI component which requires a service? Mock objects are a useful way to tackle these types of problems.  If you design your code to allow for the ability to use mock objects, you can simulate them.  In order to do this your product needs to be written clean enough that you can swap mock objects & the real objects. 

Take for example a product that relies on a SQL database. You likely have a data layer that handles your calls out to SQL.  When implementing this data layer, you can implement a mock object to simulate the SQL server.  This would allow you to do things such as simulate the results of the database or if the network is down. Mock objects are also useful at the unit testing level as the developer can isolate their component to only rely on these mock objects.  This is especially useful if they depend on another component also under

development which may not be ready for consumption yet.  If you are following test-driven development (TDD) then it is also useful for being able to write tests prior to implementing the product code.

Conclusion

Similar to developers, testers need to stay abreast with the various design patterns that can be used in development. As products continue to grow in complexity gone are the days of a tester being able to strictly white box test and black box test an application after development is code complete. Doing so will likely result in a product that is highly difficult to test and result in automation that is less robust and reliable. Instead we expect and involve our testers from the earliest stages of design both as a sounding board for designing a solution to a problem and as a source of input to ensure the product we develop is highly testable.

• Test harness
• Logger
• Common code for building test automation
• Reporting websites
• Databases for storing various results
• Machines for running tests in the lab
• Topology for your product to simulate customer deployments
• Tools for managing your infrastructure
• Tools for testing & possibly for customers
• Etc.

As you might imagine this can become quite an undertaking and in larger teams this is sometimes delegated to a dedicated lab team to support.

On top of all this work is the set of tests that your team is running & many times it also supports your development team for their development needs.  A high quality infrastructure is something that can sometimes be taken for granted, until there is a problem which can at best slow down your team’s efficiency or at worse invalidate your testing due to some bad assumptions.

Bugs are an equal opportunity employer so similar to the previous post how you need to prove your product is correct, that extends to your test infrastructure. When you find a bug are you ready & able to prove that it resides in the product code & is not a problem with your test infrastructure?  Here are a couple real world examples of both to further demonstrate the need to focus on how you are building your test infrastructure.

Will this thing ever finish?

Working on a sync engine for Microsoft Identity Integration Server (MIIS) we were investigating the ability of the product to scale.  As such a tester had instrumented some counters to track the rate that sync occurred as it ran through a couple million objects.  This process was taking a couple days, which was expected due to the scale being tested.  The tester was then leveraging the data about their rate of sync to determine the performance of the synchronization engine and then extrapolate out an ETA for the test to complete.  As he plotted this out each day the rate would drop and he would give an updated ETA.  Below is a hypothetical table for the data that was being collected.

After several days of work, we realized that every day the rate moved enough that it would never complete. Sound the alarms, we have a bug!

Now you spin up your developer & proudly point to a newly found bug.  After some investigation through it is found out low & behold yes there is a bug, but it is the test infrastructure causing it.  What is that you say?  How can my test cause a product bug? In this case it turned out to be that the way the tester implemented the counters for monitor the product were doing an expensive query via WMI on a VERY aggressive basis. This query was tying up system resources & therefore causing the slow down in the product.

Lesson learned is to be aware of the observer effect and try to ensure your test code is as unobtrusive as possible when measuring items like performance.

Test A says pass, but Test B says fail?

In another example of working on performance on Forefront Identity Manager (FIM) we were in the midst of a major improvement to performance.  We had some very talented developers on the project & as a great developer they had written some excellent unit tests to help test their progress closer to the API level.  Similarly we had some tests which covered end to end UI scenarios for performance. The developers reported great progress in improving our performance and so we kicked off our tests to see how we were looking end to end.  It so happened that our management was meeting to discuss our project progress with their managers, so they were eager for an update for performance. We kick off the UI performance tests & find that everything is timing out!

We have two very different results, so which should we tell our managers to report? We have to figure out which test is correct and why is there a difference. Well a good start is to minimize the variables, so we took the tests out of the picture by doing some manual testing.  Sure enough our manual results were inline with what the UI tests were showing,

Build it & they will come

All this to say whenever you build any piece of code, there are sure to be bugs to come from it. Teams must not view something as just test code & therefore hold it to a lower quality bar, otherwise it will quickly cost them time & invalidate their results. 

Similarly as a tester you get to build all kinds of infrastructure to help make the team more efficient. This also means you get to learn all kinds of new technology.  Need to build a new report?  What tech do you want to use to build it? Why not write it in Silverlight or WPF to learn something new?  You get to be the PM, Dev, & Tester for these types of items.  Just remember you want to test it thoroughly, otherwise there might be a bug that will bite you later.

The below picture has circulated before on how the various components of product development looked at a simple problem.

One might add to this “How the tester tested it”

Part of test’s responsibility is to prove that their feature is correct.  This includes not only that it is functionally correct but that it is actually what the customer wants\needs.

Customer Correctness - Involve your customers

I covered this in the previous post about knowing your customer, but it is worth reiterating as it is highly important.  Involving your customer as early & as often as possible will help you increase the likely hood of delivering what they really need.  There can be obstacles such as maybe you are creating something brand new so your customer has not seen anything like it before or maybe you have competitive requirements to limit disclosures.  Yet these are usually not insurmountable, at the bare minimum you can solicit feedback from co-workers on the work that you are doing.  Is what you are working on targeted at a less technical customer & all your co-workers live & write code, then what about your business admin?

Functional Correctness - Quod erat demonstrandum (QED)

A typical question in a product release is how do we know that the product is done & ready for release.  This can fill up many meetings & even books about the subject.  One aspect of this I wanted to focus on was thinking beyond the procedure of manually coming up with a set of test cases, build them, & then run them.  Instead, here at Microsoft many testers have degrees in Computer Science\Engineering and many have advanced degrees including Masters or PhDs.  As such we should borrow from our math background & think how you might prove the feature is correct (i.e. QED)

This is more aspirational, but I have seen concepts & ideas used in test to attempt to do this exact thing. As such this is one of the hard problems in test. More broadly these are collectively known as Formal Methods and this is one area which MSR has made considerable investment.

Model Based Testing

One example of how you can prove a feature is correct is model based testing.  It is an interesting aspect of how you might prove that a system is correct.  I am personally still learning more about model based testing & how we can use it in our testing tool belt when developing products.  So this is just an intro to the concept for those unaware.

The basic concept is that if you can map out a conceptual model of your product, then you can use an algorithm to walk your model which will then generate many of the test cases for you.  Just taking the time to create a model will greatly help in understanding your product and typically cause you to find questions which will then feedback into your dev & PM specs. For example, what happens if we are in X state & then Y happens?

Microsoft Research has done some work around Model Based Testing (MBT) and since has been released as a power tool for Visual Studio 2010 called Spec Explorer.  You can even use Spec Explorer to convert your understanding of the spec into a programmatic model and then when the model is explored find if there are invalid transitions that occur based on some constraints helping you find bugs in your spec & design early on.

One key consideration when building out a model of a system is the scope of a given model. If you attempt to model in detail a complex system, then you will run into state explosion where you will have way too many permutation & combinations to be useful.  Instead you should focus on modeling some specific area you are testing to minimize this problem.

Oracles

In order to verify any test result, a tester must develop an oracle to determine if the result of a test should pass or fail. Sometimes these oracles are extremely simple.  The simplest being hard coding the answer to each of their tests.  Yet this limits the number of tests to the ability of the tester to generate tests & answers to those tests.  Those answers must be maintained as the product changes over the course of development. On the other extreme is to perform complex calculations which simulate the product. The problem here is the classic of if test is testing product code who is testing this complex test code? 

MBT solves this by using the model you create about your product. Tools like Spec Explorer explicitly capture state in the model and allow for the model to return values, which can then be mapped to actual values which the system will return.  So tests generated by Spec Explorer essentially have the oracle embedded in each generated test case. But to do this, the model may have to closely reflect the algorithmic complexity of the system being tested. This can lead to high cost of model building. MBT approaches suggest to solve this by using a collection of models, including the possibility of building composite models (i.e. A more complex model built on top of simpler ones)

There are also other possibilities. Take another example of how one might leverage mathematic ideas is how you can test something as Query & Set calculation in Forefront Identity Manager (FIM).  Given a complex problem to test, the test writer needs to create an oracle or way to verify the results that the product is producing. The complexity of tests can be limited by the imagination & ability of the tester to validate their results.  Yet if you can leverage something as an oracle to verify the correctness of another item.  In the case of query if we can express a dataset in another form & translate our query against that dataset into a different form, then we can create a simple oracle.  So taking Query in FIM, we know that the query is given in XPath.  Conveniently enough XPath is typically used for querying an XML document.  So now if we can represent our dataset as XML, then we could use XPath with .Net as that oracle.

This leverages the concept of transitivity in mathematics. Basically if A=B & B=C then A=C. Or in our case if it is true that .Net XPath returns a valid search result for a XML document and we represent our dataset as a XML document, then this will give us our answer\oracle. This has the advantage of us being able to issue complex tests & leverage existing work to validate our new work.  Otherwise we would have to hard code our own test inputs & expected outputs or create a verification algorithm which could reach the complexity of the product’s algorithm. 

There are of course some assumptions & limitations here. First we are assuming that the .Net XPath of a XML Document returns the correct answer, but there may be bugs that cause it not to.  Secondly we may have data or queries that we are unable to represent appropriately. These are items the tester has to evaluate as they are determining their approach & may choose to do additional testing around the limitations of their approach.

Now that we have Query’s oracle we can use transitivity again and use Query as an oracle for Sets.  Sets makes a bunch of complex calculates on the fly. As such we can build a test case generator based on some descriptive language & then just turn around and verify that all the Sets that are calculated on the fly are correct by walking the Set memberships & checking them with a simple query.

Note: I don’t claim credit for the above solutions using in testing Query & Sets in FIM. These were great ideas that other testers on the team developed for dealing with testing these hard problems.

Conclusion

As a tester you should consider not just does your feature conform to the spec that you have been given, but is the spec & design correct to start with.  This must include rationalizing it against your customer’s needs.  Additionally a tester should move beyond just simplistic functional testing of a complex system but how they can prove the correctness of their feature.  Proving correctness should move beyond static validation of 1+1=2 and more into leveraging the testers backgrounds in mathematics and other algorithmic approaches where applicable.

Ultimately when developing your test strategy here are a couple questions you should strive to answer:

• Is this the right feature for my customer?
• To be right it must include solving a customer problem & delivered in a way that they can consume it.
• How am I proving that the feature is functionally correct?
• What are the limitations & assumptions of my approach?
• Am I testing my feature in a way to cover how my customers expect to use it?

1. Able to paste code from blog - such that readers could copy & paste code into their own editor
2. Syntax highlighting – Syntax highlight just makes things easier to read
3. Minimize scrollbars – Avoid requiring scrollbars to scroll through the code

As such, I have checked out several & the one I found to work the best so far is the Paste As Visual Studio Code add-in for Live Writer.  Ideally, I could use something not specific to Live Writer such as Copy As HTML add-in for Visual Studio, but that is currently only supported for VS 2008.

If you have a favorite tool for copying & pasting code, let me know.  One addition I would like is the ability to hide\show code in my blog posts to avoid cluttering it up too much.

Take for instance the following code:

Parameterless Constructor

If you run this through & try to serialize & then deserialize you will run into a problem just serializing this. The first error you will hit is:

System.InvalidOperationException was unhandled
  Message=There was an error generating the XML document.
  Source=System.Xml
  StackTrace:
       at System.Xml.Serialization.XmlSerializer.Serialize(XmlWriter xmlWriter, Object o, XmlSerializerNamespaces namespaces, String encodingStyle, String id)
       at System.Xml.Serialization.XmlSerializer.Serialize(TextWriter textWriter, Object o, XmlSerializerNamespaces namespaces)
       at System.Xml.Serialization.XmlSerializer.Serialize(TextWriter textWriter, Object o)
       at MyNamespace.Class1.Serialize()
       at MyNamespace.Program.Main()
  InnerException: System.InvalidOperationException
       Message=MyNamespace.MyClass`1[System.String] cannot be serialized because it does not have a parameterless constructor.
       Source=System.Xml

Xml Serialization requires you to have a parameterless constructor, so you can either add one, or switch to using WCF DataContract Serialization. This is as simple as changing the attribute on your Classes from [Serializable] to [DataContract] and then updating your Serialize & Deserialize methods. This will allow you to successfully serialize & deserialize the objects without an exception.

Serializing Read-only Properties

Next you will his the Debug.Assert which was a check to ensure we are serializing the read-only property was correctly being populated & in this case it is not.  The reason being is that XmlSerialization will automatically attempt to serialize all public properties, but WCD DataSerialization does not automatically attempt to serialize any properties.  So you need to decorate any item you want serialized with a [DataMember] tag.  This is great because now for readonly properties you can just mark the underlying private data as the DataMember.

Serializing Generics

The next problem you will hit is a SerializationException about not being able to Serialize the generic MyClassOfString, aka MyClass<string>

System.Runtime.Serialization.SerializationException was unhandled
  Message=Type 'MyNamespace.MyClass`1[[System.String, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]]' with data contract name 'MyClassOfstring:https://schemas.datacontract.org/2004/07/MyNamespace' is not expected. Consider using a DataContractResolver or add any types not known statically to the list of known types - for example, by using the KnownTypeAttribute attribute or by adding them to the list of known types passed to DataContractSerializer.
  Source=System.Runtime.Serialization

To resolve this you will need to use the KnownType attribute & there are a couple options:

1. Generically implement serialization
2. Explicitly list the known types

Generic KnownType

Generic serialization is nice as the serialize since you don’t have to decorate your code  with the various possible classes used for your generic.  To implement it you add the KnownTypes attribute to your class with a string of the method name to call.

Explicit KnownType

If you do the above then you will wind up with an exception during deserialization now since Class1 does not how to deserialize MyClassOfString. As the deserializer you will need to explictly state what types you expect.  In this case you can do it with a single attribute on the top.

Final Results

At this point the class will successfully serialize & deserialize fully. Due to the nature of my example, the step of using the GetTypes method is not required since I am doing both serialization & deserialization in Class1, but it is useful when that is not the case.

Knowing your customer

So your PM, Dev, & you have implemented the greatest feature.  Do you know what problem this is going to solve for your customer?  Do you know how they expect to use it?  If they are going to be deploying your product do you know what type of topology they will be using?

A good example of this is the Mars Rover that was lost in 1999.  One team designed their system using English units, but the other team was actually using Metric units.  I bet their tester ensured the feature worked wonderfully but there was one costly assumption that was overlooked & after a 286 day journey the probe was burned up in the atmosphere.

As a test team you will be charged with figuring out how you will be testing the product. The assumptions you make will wind up impacting your customer & their experience with your product.

Developer Bias – “It works on my box”

During the course of development it is easy for something I will call developer bias to set in.  You have been working on the product & seen it evolve over time. You can easily grow accustomed to issues or why something might work the way it does even though it is not intuitive to your customer.

One refrain that exemplifies this is a phrase you might hear from a developer of “It works on my box.”, when you report an issue with the product.

Take for example a server product which uses SQL as a datastore. During development most of the team will likely install the product all on a single box.  Who wants to maintain multiple machines if they don’t have to after all.  But customers won’t be running in that configuration & will likely have a SQL machine & run your server on a separate machine.  Does your product work in that topology?  Oh & how did you install your SQL server?  SQL server can have named instances.  Can you install to a named instance?  Does your product work when installed to a named instance?

Customer Personas – “It works great for me”

Although you may be one of the target customers for your product, are you a true representative of all your customers?  What does a teenager expect form your product vs. a stay-at home mom?  Each customer has their own unique needs & scenarios that they are interested in, you need to be aware of them and how you can cover them in your testing.

Let’s look at something such as Kinect for Xbox 360 & a feature as simple as someone waving to launch the Kinect Hub.  Seems pretty straight forward right?  Interestingly enough eHow even has instructions that are as simple as 1, 2, 3.

1. Lift your hand toward the person to whom the wave is being directed.
2. Spread the thumb away from the other fingers of the hand and hold the rest of them together.
3. Pivot at the elbow & move your hand from side to side several times

It all seems so simple, but do a sampling of people & have them wave at you.  You will find a large variance in how they actually wave to you.  Some might just slightly move their wrist, others might move their hand back & forth very rapidly.  Are kids part of your customer set?  They can wave even more differently with their hand almost sideways moving more up & down.

Conclusion

But I am just a tester, isn’t PM the one that talks to customers?  They do, but do they understand the assumptions you are making in testing?  Have you used the many resources you have to find out information from your customers?  At Microsoft there are several avenues to get this information: TechNet Forums, Pre release programs through connect.microsoft.com, various conferences, external user groups, internal distribution lists, MVPs, etc.  Work with your team to reach out to your customers, I am sure they will be more than happy to help you build a great product that will help solve their problems.

Knowing the code & sometimes ignoring it

So understanding how your product works is a good start, but as mentioned earlier a tester needs to have breath & be able to go deep.  Part of that depth is understanding the code behind their product.

There are a couple examples of why you will need to have this depth:

1. Designing your automation frameworks
2. Debugging your product

Designing your automation frameworks

While development is designing & implementing the product you will need to figure out how you are going to test the product, while also being involved in the product design itself. Understanding how the code is implemented is going to help with designing your automation.  For example if you have a typical 3 tier application, how is the UI talking to the middle tier?  Are those APIs public?  Do they give you what you need to write automation?  How can you extend them for what you need?  Is there a better way to design the product to make it more testable?

Debugging your product

Part of being a tester means you are one of the first customers of your product.  You are developing tests against a moving target as developers are implementing new features & making changes.  One day a test may be passing & the next it fails. Is it a test issue or a product issue that is causing it to fail?  Your product may not have much manageability yet so there are no good events in the log that you can use.  The only thing left is being ready to strap a debugger to the product & step through the code to understand where things are going wrong.  If it is a product change that broke your tests you may need to understand how to update your tests to work again. Oh & when you are done, you’ll want to file a bug to improve the manageability of the product. You are doing something a customer would be doing and they won’t be able to strap a debugger to it. You’ll want enough information in the logs that ideally a customer could resolve the problem.

Ignoring the code

So you now know your product’s code inside & out.  You are able to debug & even provide your developer with proposed fixes.  You might even have helped write some of the features in your product.  Now forget everything you know about the code.

What?! You might say, I worked so hard to figure it all out. Yep & if you are not careful you may make some of the same assumptions that the developer did when they wrote the feature.  That means you run the risk of designing your test cases around the code & missing some juicy bugs.  You need to step back & challenge any and all assumptions that you might be making.

Along with challenging the assumptions made in the code, don’t forget to challenge & look at everything from the customer’s eyes.  Does the instructions we provide in the UI make sense to the end user or are we using 3 different terms to reference something because it’s name has changed over the course of development.  One powerful tool here is to swap features with another tester & try each other’s area out.  If they can’t figure out how to use your feature, don’t dismiss it as they are new & don’t understand the product enough.  They are likely stumbling over some areas that a real customer would & since you are neck deep in your feature & code, of course how to use it is obvious to you.  It likely won’t be obvious to your customer, so look for how you can improve that also.

Problem introduction

Take for example a feature called Query in Forefront Identity Manager (FIM).  You start out on the feature & learn that it takes a query in the form of XPath & then returns the results of that query.  You have some standard set of operators that you expect to find such as =, >, <, and or, not along with some functions like starts-with, ends-with, dateTime, etc.  The specific operators are not the important component here, but if you are so inclined you can view the details here.

Knowing your product

As a tester you might be prone to take this problem & run with it to create a bunch of test cases around the various combinations of operators & data values.  Initially you find several bugs & you are keeping your developer pretty busy.  You even think about some of the various buckets of tests like boundary, localization, globalization, performance, stress, security, etc.  Are you now ready to declare your testing complete?

One of the things a good tester needs to do is fully understand the problem space they are working in & also how it may be changing over the course of the product lifecycle.  For example in this case the design of FIM leverages the XPath filters of Query in a feature called Sets.  These Sets are then used in another feature called Management Policy Rules (MPRs). MPRs are core to the product in determining both if you are allowed to do something.  These MPRs also can trigger another feature Workflows which can then perform various actions in the product.

Yikes! You have just gone from a tester of a single feature to something that is deeply involved in 4 different features which are ultimately core to your product.  And note that these features are likely covered by different developers, testers, & PMs.  So you may not be responsible for these other feature areas, but you will need to know & understand how your feature interacts with them.  Even better if you are in active development the design may be rapidly changing as your team reacts to requests in future milestones, so what you knew yesterday may not be true today.

What’s the Impact?

With knowing your product you can now start to evaluate the level of coverage you have & where you might want to add more coverage.  One example of how you will need to use this knowledge is justifying the impact of a bug.  Let’s say you found a bug in Query which when searching for “Smith” exclusively returns all people with the last name of Smith, but also Smithers are returned at the end.  It is the day before you ship the product & the question will invariably be asked: “Can this bug wait until the next patch, service pack, or release?”  Or put more simply, “What bad things will happen if we don’t fix this right now?”

Now your knowledge of the product is key. When the feature was originally designed it was just used for searching a list of users or groups, so returning a few extra users would be no big deal right?  I mean who would page through the list of all Smiths in a phone book to see that there are a few extra results at the end?  But wait remember from above what you learned about the product & understand where else your feature gets used which is granting rights in the system & kicking off workflows.  So now this bug has increased in importance since you would potentially be granting rights to users & starting processes which are not intended.

So as you can see as a tester you better know your product almost as good as anyone else to help ensure you have the appropriate level of coverage & you don’t get the luxury of just knowing a single feature area but need both depth & breath across features.  If you want to really know your product then no better way than to test it.

This series of posts is not by any means authoritative, but instead meant to give you some food for thought about the types of challenges you can expect to face. So if you are considering a position as a SDET with Microsoft & wondering what you might be able to expect, or just a customer curious about the role of a SDET at Microsoft I hope you find the coming posts useful & informative.

In the following series of posts, I will cover the following things that test must do to help develop a successful product.

1. Know your product – You may be assigned a couple feature areas, but many times just knowing your own area is not enough to ensure good test coverage.
2. Knowing the code & sometimes ignoring it – Improving test coverage is about knowing how the product code works & at other times ignoring how it works.
3. Knowing your customer – Your team has developed a great new whiz bang product, but do you understand the problems that it will solve for your customer & how they will use it?
4. Proving your feature – It is one thing to test your feature, but how can you prove it is correct?
5. Impeccable Infrastructure – Your tests are only as good & useful as the infrastructure that you use to run them.
6. Ensuring Testability & Influencing Design – Your team is planning the next big thing, using some awesome new technology, but do you have any idea on how you can actually test it?

So feel free to follow along & let me know if this information is helpful to you or there are other questions you have.