Is a tester ever done with testing (Part 2)
Hope you got a small glimpse of a my world as a tester in my previous post.
This post continues to shed light on more specifics of testing I have witnessed my team do and how it contributes to the we-will-never-get-done attitude I mentioned in my previous post.
As you might already know, we test C# IDE in our team. Testing C# IDE, which is just a part of the whole Visual Studio product (VS for short, from now on), basically boils down to testing a handful of components that go into building all of the C# IDE features into the product. While it is obvious that we are most of the times just testing these components, we also have to test for the following real world/customer scenarios to make sure the IDE features work just great.
- Side by side scenario: This refers to having an older version of VS on the same machine with side by side newer version (which is of interest to us for now since it is this newer product that we are going to ship this time) on the same machine. This scenario calls for validating that
· the newer version of VS installs fine side by side with an older version of VS.
· the older and the newer version continue to work as expected in this side by side situation
· the uninstall of any one of the products leaves the remaining one(s) in a good functioning condition
- Vertical integration scenario: This refers to having a higher order SKU of VS side by side with a lower order SKU of the same version of the product on the same machine. SKUs of VS can be
o role based (developer, architect, tester) or
o the one that has all the roles rolled into it (Team suite) or
o simple express editions that cater to very lightweight, non-professional and hobbyist developers or
o the ones that cover core grounds of development like Professional (PRO), Standard (STD) and the one for office (VSTO)
More on SKU and their differences for VS2005 can be learned here and here.
Testing this scenario validates that
· a lower order SKU and higher order SKU can be installed side by side
· both lower and higher order SKUs work as expected
· the uninstall of one will not affect the remaining SKU in any unexpected way
- App Isolation scenario: All the express editions of VS (C#, C++, Web and VB) can all be installed side by side on the same machine and testing this scenario aims to validate that
· all the express editions function well as expected side by side
· uninstall of one or more of the express editions leave/s the remaining one(s) unaffected
Testing that the product of interest works fine in the above scenarios requires us to run our tests in the above various combinations of product configurations. For example – we test to make sure that C# IDE works fine on VS2008 with VS2005-SP1 and VS2008 on the same machine, that the C# IDE works fine on C# Express on a machine with C# express and web express with web express uninstalled later on.
Apart from these SKUs, VS gets translated into 8 more languages other than English which are Simplified-Chinese, Chinese-Taiwanese, Japanese, Korean, French, Italian, German and Spanish. Our tests that validate the correct functioning of C# IDE should also run on these language products to make sure our localized version of IDE works on par with the English one.
There is one more interesting scenario in which the product is used and that is “Globalization” scenario. This refers to using English VS on top of a non-English operating system (OS). This is to cater to those customers that belong to certain parts of the world where English is pretty commonly used along with their native languages and people there are expected to use our English VS products on their native language version of OS’s. Testing this scenario validates that
· IDE is Unicode-compliant
· IDE is capable of handling Unicode and left-to-right or bi-directional languages as inputs in code editor, dialog input boxes etc
· IDE is capable of receiving inputs from non-English keyboards, etc
Examples of this testing include – making sure that the C# IDE in VS works fine when installed on a machine with Arabic Windows XP etc
Machine architectures also play a role in this whole of test matrix and we need to test on x86 and x64 platforms to validate we are good to go on these as well.
Added to this whole set of things are the different OS’s themselves – like Windows XP, Windows 2003 Server, Windows Vista etc as they ship one or more versions of the .NET framework, an OS component that the VS is very closely bound to.
One last interesting element in this already huge test matrix is that our C# IDE components are invoked from various other clients within the VS –for example, C# intellisense has to work in immediate window in debug mode, most of the C# IDE features have to work in a web project or website project also, etc.
One good thing in all this seemingly infinite set of possible test scenarios is that we can still use the same set of tests over and over again to validate our part of the product without having to author special tests for any of the configurations. It is, however, obvious that just running the already written tests over and over again on these hundreds of configurations requires time, effort and resources.
The important point of view I gain as a tester, which is easy for a developer to miss, is that a feature needs to be developed once but needs to be tested in hundreds of ways before we can sign off on it as good enough to be shipped. In doing this, we testers tend to become the most expensive part of the whole product building process that I briefly described in my previous post.
Hopefully I provided a small insight into a tester’s world by describing all the elements one is required to go against while assuring the quality of the product as being shippable.