4) Always write down (or at least mention) the first implementation that comes to mind, even if you get the feeling there might be a more efficient way. This includes handling all your special cases which you can add once you have basic implementation.

Why?

You’ll show that you know how to solve the problem. A lot of times there’s more efficient and elegant solutions than the first one that comes to mind. After implementing the simple one you can mention that there might be another and keep thinking about it (remember brainstorm out loud!) or you might get asked if there’s a better solution anyway.

Example:

int total = 0;
for (int i =0; i < a; i++)
{
    total += b;
}
return total;


5) Run through your test cases and see if your algorithm holds, specially run through the most interesting ones. This is where they become useful!

Why?

You want to validate your algorithm; that was the whole point of having a list of scenarios earlier. If your test cases are good enough you might also pick up bugs on your algorithm and it’s always a plus if you find your own bug instead of having your interviewer point it out and having you figure out where it is.

Example:

Depending on the complexity of the algorithm I like to use tables here as well when running through loops (for this example this would be overkill and a waste but just to illustrate the point here)

For the scenario where a =2, b =5 and expected result = 10

 If we tried with the negative test case, a = -1, b = 1, expected result = -1

 At this point you can see that it doesn’t handles negative very well.


6) Fix any bugs you might have found or make any optimizations (which might involve rewriting the whole thing so if that’s the case repeat the last step after you’re done).

Why?

If this one isn’t obvious as to why you’re probably looking for work in the wrong field.
Having bugs in your algorithm is not necessarily a bad thing so don’t get discouraged if you find one or if your interviewer points it out for you. Finding and fixing bugs if part of the software cycle so only panic if you still can’t see the bug after you get several hints of where it’s at.

Example:

The original code didn’t handle negatives so we’ll modify the code to handle that:

int v1 = Abs(a);
int v2 = Abs(b);
int negativeCount = 0;
int total = 0;

if (a < 0)
{
    negativeCount++;
}

if (b < 0)
{
   negativeCount++;
}

for (int i =0; i < v1; i++)
{
    total += v2;
}

return negativeCount == 1 ? 0 – total : total;

Finally, be ready to explain any choices you made in your implementation if your interviewer has any question. The cliché there’s no right or wrong doesn’t completely apply here (as obviously code full of bugs is wrong) but other than that you should feel confident on your code and be able to explain why you went with that implementation. If you get a question like that don’t immediately think that there’s something wrong as this isn’t necessarily a trick question. It might just mean that you have a different implementation than what your interviewed expected and he’s curious as to your design choices and he wants to get more insight as to how you think (which is a good thing!).

https://channel9.msdn.com/Shows/C9-GoingNative/GoingNative-5-Inside-the-Visual-C-ID-Meet-Raul-Prez

Happy Holidays to all of you out there who are in some sort of holiday state. If not, then happy holidays anyway from Diego, Charles, C9, and VC

We don't cover software testing—the job discipline—often enough on C9. We aim to change that starting now.

A friend of Diego's on the VC++ team, Raul Pérez, is a software developer from Puerto Rico who works in QA for the Visual C++ IDE team. He writes tests to make sure the very-front-end of the VC toolchain—the IDE and its design-time compiler infrastructure—works as expected.

There's a lot going on when you type characters into the VC++ editor. What happens, exactly? Why? What types of things can make Intellisense fast? What types of things can hinder the performance of the IDE? How does all of this magic happen? There's a compiler involved in all of this. It's not the front-end compiler (cl), but it is a front-end compiler and it compiles your source into data that's stored in a local DB for design-time use by Intellisense, Go-To-Definition, Syntax Coloring, Reference Highlighting, Auto-Completion, etc... All of these things are part of the set of IDE features that make Visual C++ visual... So, meet Raul and learn a thing or two about how the IDE works under the covers and how the system has evolved over time.

Abstract:

Ever had a situation where you saw a test failing but since it didn’t immediately repro the failure got dismissed and then later the issue resurfaced and it ended up being a serious issue? We’ll explore as to why even though most of us agree we shouldn’t not ignore failures people still do along with some real case studies of occurrences of this that I’ve come across as a tester in Visual Studio. Finally we’ll talk about when it’s “ok” to dismiss a failure and ways to influence the culture around this.

Section 1: Introduction

Does this scenario sound familiar?

• Test fails
• First thing whomever looking at it does is to run it again
• Test passes
• If there was a bug it’s closed at not repro
• If it was on a test run it gets ignored as pass on rerun
• Real bug keeps lurking and resurfaces later

If it does it also probably frustrates you to no end. After all, tests are written for a reason and when they fail it usually means something is wrong. In theory all this sounds right but in practice it might not be the case. So why is it that we’re so far apart between what is should happen and what actually happens? From experience I can summarize the top reason for that in one of these 4 buckets:

• Everyone is on a tight schedule – This means that when there’s a business decision to be made you have to take some calculated risks and if a test is only failing sometimes that’s something you might justify living with.
• Investigations are unattractive work – Some people love digging into problems and figuring them out. Others view it as something that isn’t fun or something that they rather not do. I’d even go as far as say that some people think that doing test investigations is beneath them. Bottom line is that it isn’t glamorous work and some investigations can be very tedious.
• Lack of manual repro must mean that the user scenario isn’t broken – This falls under the blame the test category (which we’ll cover in more detail later). It’s very easy to blame test automation (which effectively translates to its someone else’s bug) when you’re dealing with a hard to repro bug.
• It’s easier to add a workaround in the test than to find the root cause – Sometimes people will agree that there’s a bug (albeit begrudgingly) but they’ll argue that working around the issue is easy and cheap so the test should be doing that instead.

Section 2: Culture around test failures

One big problem as to why even though people agree with the statements in the previous section is how tests (and by association) the testers are viewed and valued in an organization and more importantly how much does that organization value testing.

There are two very distinct lines of thought when talking about tests and as a tester you should always be pushing for the later:

• Tests are just things that need to run and pass before I check in my code changes.
• Tests results paint me an accurate picture of what the quality of my product is.

If you’re stuck around people that have the mindset of the former you’ve probably seen both of these:

• When a test fails the first thing a good number of developers ask is, “Is this a flaky test?”

The first thing you should ask yourself when a test fails is what new bug got introduced in the product and how do we fix it? Not if the test itself is full of bugs.

• The Guilty until proven innocent mentality

This is a variation on the last statement. For a lot of developers until you can prove and explain a product bug it’s considered a test bug.

Both of these paint a bleak picture where the overall mentality ends up being “It’s never my fault, it’s always yours” and its sibling “It’s someone else’s bug”. This has the unfortunate side effect of affecting how testers are valued and these are statements I’ve heard that make me roll my eyes because they’re wrong and most of the time unfair and untrue:

• “This is a crappy test so I just keep running it until it passes.”
• “My code is fine, your test never really finds issues it just fails randomly.”
• “Can we fix the test so that it’s more resilient to bugs?”

Section 3: Common issues on the product and on the test side

So how do we fix all this? First thing we need to understand is why there are failures that feel random and that there’s a multitude of issues on both the test and the product side that can contribute to this. Here’s what I’ve seen cause the most instability with test results and it’s worth noting that as everything around software there’s enough blame to go around and both sides (development and testing) have their own share.

On the product side of things:

• Issues that only manifest themselves on the application first load

This one is self-explanatory, these are issues that only happen on cold boots or the first time ever the application runs on a machine.

• Race conditions

One of the “fun” parts of dealing with threading. When there’s multiple threads sharing resources assumptions are usually made and a lot of times they can be wrong and cause race conditions when the bugs only manifest itself when one thread gets there before another and one of them assumed there was going to be some sort of order.

• Reentrancy issues

Another threading issue, one of the common causes of these is when reentrant functions presume resources are thread safe while they’re not necessarily completely thread safe.

On the test side of things:

• Dirty test assets

Depending on what your application consumes your assets might not always be clean. One good example here is testing a test editor where you’re opening an existing file and doing some edits. The first time the test runs it might be pristine but if you don’t do things right or cleanup correctly subsequent runs might run against a different file.

• Dirty application settings/environment

When the test depends on the state of the machine and/or the application and it doesn’t ensure that everything is on the expected state you’ll see different results depending on where tests are run.

• Lack of synchronization

Not every action finishes instantaneously so a lot of times you have to wait until that action is finished to validate your result or to perform other actions that depend on the previous one. A lot of times the synchronization is happening as part of your test API’s but you shouldn’t take anything for granted and always confirm that the correct synchronization is happening.

Section 4: Case studies

Let’s talk about actual examples of issues I’ve seen while working on Visual Studio over the last few years that illustrate some of the issues I previously mentioned. I’ll go over one common test bug and two interesting product bugs I’ve had to deal with.

Section 4.1 – Blame Your Tester

This is something I’ve seen way more times than I’d like to admit on multiple incarnations. Can you spot the bug in this algorithm?

OpenVisualStudio()
CreateNewProject()
RequestIntelliSense()

The problem here is that after you create a new project IntelliSense is not immediately ready (is this reminding you yet of the synchronization issues we talked about before?). It’s a very easy issue to fix as long as you have the right tools and you actually spot the issue. Now this is how it’s usually solved, does it sound right?

OpenVisualStudio()
CreateNewProject()
Sleep()
RequestIntelliSense()

So now we wait and then we perform the action. Issue solved right? No, and I can’t express this enough. If you see a sleep there better be a good explanation as to why it was the last resort or sooner or later you’ll hit the same issue. Sleeps are non deterministic, for all you know it might exit way earlier than the time you actually needed or even worse it will work most of the time and fail other times.

What should happen instead is something like this:

OpenVisualStudio()
CreateNewProject()
WaitForIntelliSenseReady()
RequestIntelliSense()

In which you have some good deterministic synchronization that waits just the exact amount of time and won’t introduce random failures.

Conclusion: Synchronizing the right way makes a world of a difference yet it’s one thing that’s easy to overlook. When you end up with cases like this in which you hit the worst possible scenario of working most of the time but still failing in some occasions the statements we talked about on section two made by developers are more than justified.

Section 4.2 – Death by a Thousand Cuts

This is an example that perfectly illustrates the “it’s someone else’s bug” attitude that we mentioned previously. It also caused me a lot of grief which you’ll see soon enough why. So to start off here’s the failure:

Before running tests our team does a couple of pre setup steps which ensure that we’re running on a clean environment. Remember that pesky issue about dirty environment we mentioned earlier? This is one of the ways we deal with it. As part of this one of the steps is to load Visual Studio and reset the settings. Now, we don’t want things to be hung so we set the timeout for this step to 5 minutes which is more than enough. All of the sudden this starts failing and everyone says it has nothing to do with them.

Now here’s where the grief part starts, these are actual parts of an email conversation I had with one of the developers investigating a failure:

Me: I’m concerned about is that I’m not seeing that issue in main (or elsewhere) and you mentioned that you hit it multiple times so if it is an issue it’s coming from the branch and it should be investigated there.

Developer: I am only looking at the debugger issue. If it fails on official runs the owner will need to take a look. I doubt that it is a product issue

Me: If you had to rerun it multiple times that would be something you should investigate as that might mean we’ll see the same failure on official runs. The timeout of that is 5 minutes, if a call to reset settings is taking that long there might be some perf regressions or other badness here.

So this already isn’t going to well, over the next several weeks I start getting emails from a lot of different people about the test failing with the same common cause that timeout. At this point I think I was close to testing how sturdy my office door was by ramming my head into it quite a few times.

As usual though, I had keep fighting the good fight as I knew that the root cause here was a product issue and something that should get addressed. So fast forward 3 weeks later and after enough investigation was done there was a broad email from a developer manager high along the food chain acknowledging that we indeed had an issue and that we had multiple people working on it. Score one for testers standing up for the tests!

Conclusion: The root cause for this failure ended up being multiple bugs on different packages that caused significant slowdowns that would only manifest themselves the first time the application was loaded. Interestingly enough the two tests that were hitting this, which of course came under fire and I had to fight off a lot of requests change in order to add workarounds, were not actually even going through the scenarios that were broken. It just so happened that their setup routines was uncovering issues that would have been otherwise ignored.

Section 4.3 – I Called Dibs!

This was an extremely interesting case which yielded some big changes which I’ll talk about later. Is there anything wrong with this test code?

OpenVisualStudio()
CreateNewProject()
Build()

It turns out everything on the test side was correct, however under the wraps there were a lot of things going. After a project is created there’s a bunch of background operations that happen in order to be able to deliver the full user experience including the work IntelliSense needs to do. In this case particularly both the IntelliSense engine and the build system have different builds going on (both using their own components).

So far so good, however both of them need to ask the build manager for some project specific data. So what happens when the build manager is busy? It turns out the way it works is that when there’s a build in progress and someone asks the build manager for information it tells them sorry I’m occupied right now, could you please ask me later? Now if any of the components is greedy and presumes that they’ll get everything when they ask for it they’ll eventually run into issues which is what ultimately happened here.

Conclusion: The root cause of this ended up being that both of the consumers needed to handle the retry calls graciously and neither of them was so depending on who started first we’d have a failure coming from the other component. As I mentioned earlier, this turned out to be an architectural issue that needed to be changed with both components understanding that they might not always have priority and that getting told to wait is a completely valid response.

Section 5: Promoting change

So all of this is great information but you’re still not sure as to how you can make a difference and change the culture around you? For starters you should always be thorough and be able to explain with confidence what’s happening (even if all you have is a working theory). Here’s three things you can do that will help your argument on failures being product bugs:

• Run the test on multiple iterations

Don’t give up when it doesn’t repros immediately, as we have mentioned multiple times just because it passes on rerun doesn’t means everything is good. Most drivers have options to run a test multiple times so abuse it.

• Restore the machine to a clean state

This can be tedious and annoying as its very time consuming but sometimes it’s the only way. If possible (and virtualization helps with this in a big way) you can restore the machine to the same state it was before the test was originally run. You don’t always have to go that far though, every application has specific things that they do the first time they run (add registry entries, save setting files, etc). If you know your product well you should be able to manually get the machine back to that state.

• Cleanup the test assets and environments before and after the test

This probably should go without saying but if you eliminate as many variables from the equation as possible it makes it hard to argue that there are issues with the way the tests are doing something.

Now, there are cases when you have to just cut your losses and accept that the test failed without having a good explanation for it. Even when that is the case my previous statement still holds - always be through and explain things with confidence. Two very important tips in this area:

• Above all perform due diligence before dismissing it outright

If you’re dismissing your test failures without exploring all possible options then you can’t expect others to respect your tests which correlate to how much respect they have for you as a tester.

• Use your judgment and prioritize

Some tests are more important than others and as much as it pains us to admit it most bugs will get fixed but some bugs will not. If it’s something that you consider extremely important go the extra mile but if it’s something that you can probably live with due to the rarity and the severity of the bug your time might be better spent elsewhere. You know your product and your features so trust your judgement.

Section 6: Conclusion

If nothing else there’s one thing that I hope was ingrained in your brain from this:

Pass on rerun and not repro resolutions are unacceptable.

If that’s all you got away from this then all this time was already more than worth it yet if you also got all these then you’ll fully appreciate testing (and your testers) even more.

• Failures are bad, if they were acceptable then what’s the point of having a test for it?
• A lot of times flaky tests are just reflecting a flaky product, don’t be quick to blame the tests.
• Until you can prove that there’s a test issue presume that it is a product issue.
• Workarounds are a sign of a buggy product, if you have to work around something it probably means your customers will have to as well.
• Even when it is a test issue, get to the root cause of it.

When used correctly code coverage can be a great data point to assess your test quality and test holes. What a lot of people don’t realize though is that code coverage doesn’t actually means good tests or even that all the areas that are covered are actually tested correctly. There are several things you should keep in mind when looking at code coverage numbers which we’ll look at here.

                If you’re already familiar with code coverage just browse over the code samples (as I’ll reference them later) and skip to the next section. I’m not going to go into details here but rather define the basic things one needs to know and show a small example using the MsTest code coverage tool in Visual Studio. The first concept with code coverage is blocks which represent sections of your code. Take the following application as an example:

01     class Program

02     {

03            static int Main(string[] args)

04            {

05                   int value;

06                   if (Int32.TryParse(args[0], out value))

07                   {

08                         if (value >= 0)

09                         {

10                                return FuncA(value);

11                         }

12                         else

13                         {

14                                return FuncB(value);

15                         }

16                   }

17                   else

18                   {

19                         return 0;

20                   }

21            }

22

23            static int FuncA(int value)

24            {

25                   return value + 1;

26            }

27

28            static int FuncB(int value)

29            {

30                   return value - 1;

31            }

32     }

That application contains thirteen blocks but for simplicity I’m going to name the major ones

·         Line 6

·         Line 7-8

·         Lines 9-10

·         Lines 13-14

·         Lines 24-26

·         Lines 29-31

The simplest way of visualizing a block is a continuous block of code that executes some code. Notice for example how the declaration of value at line 5 doesn’t counts as a block. In general you’ll never really worry about where blocks are since code coverage applications do that for you. When a test executes some blocks will be hit and the percentage of blocks hit will be your code coverage numbers.

I added this simple test to my project:

        [TestMethod()]

        [DeploymentItem("ConsoleApplication3.exe")]

        public void MainTest1()

        {

            string[] args = new string [] {"0"};

            int expected = 1;

            int actual = Program_Accessor.Main(args);

            Assert.AreEqual(expected, actual);

        }

After running the test I get the following numbers:

And Visual Studio colors the blocks to show me what’s covered (blue) and what’s not covered (orange):

A good test and a bad test that have the exact same coverage

Code coverage will only show you which portions of your code were executed when running a test but it will not tell you if those portions of the code actually behaved correctly. That means that code coverage is somewhat reliable in order to say things like my application runs with the following input set. It doesn’t tell you however how resilient or buggy your application might be. On the test side it doesn’t means that your tests aren’t a pile of garbage, just that they touch certain spots.

Take the same simple test we used above (and keep in mind this is a simple application and this is an extreme case). If I removed the last line of the test (the one that does the verification) the code coverage numbers will come back exactly the same. Let’s take that a step further and run this test:

        [TestMethod()]

        [DeploymentItem("ConsoleApplication3.exe")]

        public void MainTest2()

        {

            string[] args = new string[] { "0" };

            Program_Accessor.Main(args);

            args = new string[] { "-1" };

            Program_Accessor.Main(args);

            args = new string[] { "a" };

            Program_Accessor.Main(args);

        }

There are no verifications whatsoever on that test but if you’re just going on code coverage you just wrote the perfect test:

I have to add a disclaimer here, while that’s the worst test you can write when doing functional and correctness testing it has its value if you’re running stress testing as it’s touching all of the code.

Background work items

Here’s one of those cases where things get tricky. If you’re testing a more complex app chances are it’s a multithreaded app and there’s some work going on in the background most of the time. Here’s an actual example I ran into a few weeks ago. I was running some tests for one of our language services and here’s how the test looked like:

1)      Open a file

2)      Wait for the file to be colorized in the editor window

3)      Verify that it’s colorized correctly

Keep in mind that code coverage is per assembly and the assembly contains the whole language service which includes way more than just colorization. I had run these tests before and the code coverage was around the 30% range but this time it gave me around 80%. You can imagine my surprise when the exact same of tests gave me 50% more coverage when there had been no changes to the code base. How did that happen?

Let me briefly explain here how a language service work (which will give you a hint of where the extra coverage comes from). When you open a file Visual Studio loads the language service associated with that extension. Most language services start by parsing the file in order to find the classifications, and then run other components for more complex information (outlining, intellisense, etc). All of these, including the colorization, are done in separate threads in the background after the document is loaded.

The difference here between previous runs and this one is that I had a bunch of other things opened in my machine that were consuming resources and thus slowing down the test. Usually the tests are pretty fast so as soon as colorization was finished the test exited. In this case that extra time that the test was running allowed some of these other background operations to complete. Easy way of proving this theory, add a fourth step to the test that sleeps for 10 seconds. Every single time I ran the test with that it hit the 80% mark.

The one take away here, if your code coverage is hitting more blocks than the scenario you’re actually testing its most likely that it’s due to background operations and your total coverage is bloated.

So what does code coverage data actually tells me?

The one thing that’s absolute about code coverage is that if you’re not getting code coverage in one area it’s simply not tested at all. You can take this information to:

a)      Add more tests for that area.

b)      Figure out which edge cases you haven’t tested yet.

c)       What portions of the code you’re purposely not testing.

One additional note to that last point, while having 100% code coverage looks nice it shouldn’t (and usually isn’t) a realistic goal. A well coded application will probably have some redundant fault tolerance that it’s expensive to test on a regular basis. For example let’s say the application queries some services which can rarely crash. A good application will be resilient to this but testing that could be expensive (say for example you would have to go and corrupt some memory in order for this to happen). There is a point of diminishing returns here.

Conversely you can use the data from the things you are covering to:

a)      Reduce code bloat by eliminating duplicated tests - Yes there can be such thing as too many tests.

b)      Ensure that your mainstream scenarios are covered – This is particularly useful when you’re evaluating the coverage on a subset of tests that you want to prioritize versus running your full set of tests.

c)       Optimize your test bed – If running your tests is expensive you can use code coverage as a supplementary data point as to add the minimum number of tests that give you the maximum amount of coverage.

The key point here is that code coverage is a very important part of testing in the software cycle however it should be used as a tool rather than a primary data point to make decisions. The numbers should always be analyzed and the conclusions should come out of that analysis not out of the numbers themselves.

I was working with some pixel values the other day and I had this problem; I’m getting an RGBA value from my object model but it returns it on a string format (ie: 0, 1, 1, .5). For this specific task I only cared about the alpha value so I figured writing a regular expression for it should be the easiest way to do this and I was in for a fun surprise.

So I started out with this:

\d,\s\d,\s\d,\s\d

Wrong!! – I forgot one cardinal rule about regular expressions, they work against string content so a decimal will not be considered part of a number. So long story short what I thought should be a simple regular expression turned into a scary one (at some point I should have given up on it and used String.Split but where would the fun in that be?). So here’s the final version of it:

\d+(\.\d*)?,\s\d+(\.\d*)?,\s\d+(\.\d*)?,\s(?<alpha>\d+(\.\d*)?)

A breakdown of that expression

\d+(\.\d*)? – This will pick up my values, it will pick up both 1 and 1.0

,\s – This is the separator which in my case is always a comma followed by a space

(?<alpha>\d+(\.\d*)?) – This is the same as the first part except that I defined this group as alpha so that I could pull the value. I

And a snippet of the code where it would be used (I defined groups for all 4 values here:

Similar expressions can be used for coordinates (both 2D and 3D)

2D : (3.1,23,5) =  \((?<X>\d(\.\d*)?),(?<Y>\d(\.\d*)?)\)

3D : (3.1,23,5,90) =  \((?<X>\d(\.\d*)?),(?<Y>\d(\.\d*)?),(?<Z>\d(\.\d*)?)\)

I had just finished implementing some formatting tests for a new language service so I ran them to see if they were doing what I wanted them to do (on a side note is testing the tests considered recursion?). While one of the newer tests is running I hear some odd beeps from my computer. I'm always listening to music while I work so at first I was wondering if that was just part of whatever song was playing. Kill the music and run the test again this time with my speakers way up and there were the beeps again. I manually did what the test was doing to make sure it wasn't something weird from the test library and confirmed that it was indeed a bug in VS.

At that point I freaked out a lil bit since it was a Friday night and most everyone had left already plus we were locking down on code changes but I did find someone still around to debug the issue with and fix it. So what was the problem? The test was simply opening an empty file and typing this:

int foo()
{
     int a = 0;

And on the first letter of each "word" IntelliSense was trying to kick in and failed (the failure was expected) so it beeped. The bug was that on scenarios like the one I was testing it shouldn't have even tried to kick in. Now you can imagine how annoying this would be on a real production environment if your computer was beeping for the first letter of each thing you typed...

Even though I noticed it while running a test there would have been absolutely no way we could have caught this through our automated tests since we don't test for sound. Lesson here never forgo doing manual testing because you have good coverage on your automated tests. At the very least sit down every once in a while and watch your tests run and make sure no "bad" behavior that's irrelevant to the test purpose (but important to your user) is going on.  

delegate void SetControlValueCallback(Control control, string property, object value);

void SetControlPropertyValue(Control control, string property, object value)

{

       if (control.InvokeRequired)

       {

              SetControlValueCallback callback = new SetControlValueCallback(SetControlPropertyValue);

              control.Invoke(callback, new object[] { control, property, value });

       }

       else

       {

              Type t = control.GetType();

              PropertyInfo p = t.GetProperty(property);

              p.SetValue(control, value, null);

       }

}

The problem with toolstrip controls is that they don’t really inherit from the control class so you can’t call Invoke on them. The trick is to call invoke on the Owner control while still passing the toolstrip control to the callback (if you pass the owner control to the callback you’ll get some funny results like your whole menu bar being disabled…). Here’s the code snippet for that:

delegate void SetToolstripValueCallback(ToolStripItem toolstripItem, string property, object value);

void SetToolstripPropertyValue(ToolStripItem toolstripItem, string property, object value)

{

       if (toolstripItem.Owner.InvokeRequired)

       {

              SetToolstripValueCallback callback = new SetToolstripValueCallback(SetToolstripPropertyValue);

              toolstripItem.Owner.Invoke(callback, new object[] { toolstripItem, property, value });

       }

       else

       {

              Type t = toolstripItem.GetType();

              PropertyInfo p = t.GetProperty(property);

              p.SetValue(toolstripItem, value, null);

       }

}

Let’s start with the basic definition of each of these scenarios:

Unit Testing – Unit testing is testing directly at the most granular level. If given a method that takes two values and return a positive result. Does the method fails (crashes, throws an exeption, etc) if either of the values is null or invalid? Does it return valid results given a specific set of values? Does it fails if given an incorrect set of values?

Component Level Testing – Similar to unit testing but with a higher level of integration. The big difference here is that the testing is done in the context of the application instead of just directly testing the method in question. Take the same example as the method above, say that the method is invoked through a keyboard shortcut after your application opened a document and after it executes something is displayed on the application. While testing at the component level you would have the application open and the document displayed but you would be testing the method and evaluating it’s result (without taking into account the shortcut or what gets displayed on the application afterwards). For the most part this is the kind of testing we do when there are test hooks inside your product code.

UI Testing – This tests the whole end to end scenario (which simulates exactly what the user would be doing). This (for the most part) goes through all the mouse clicks and keyboard presses the user goes through to get an action done. If we extend the previous example this would be invoking the same action except that this time instead of using any test hooks it would be using the keyboard shortcut and instead verify the actual displayed results on the application.

What method to use?

The question that first comes to mind for everyone, since UI testing is the most complete and closest to the user why not simple do UI testing exclusively and be done with it? Going through the UI will execute the action at every layer and that in turn will call the individual methods so why not simplify things? The issue here is that for extensive scenarios you have to pay the cost of loading the application and going through all the actions every single time (which depending on your application it can take several minutes per test).

After hearing the opposite comes to mind, so if doing UI testing is so expensive why not simply do Unit Testing which is pretty cheap since it only needs to invoke the functions for the multiple scenarios? With this approach you’ll know that all your methods are working properly but you’re not testing the integrations of the features and the product as a whole.

That leaves us with the third option; component testing seems to be a pretty good happy medium? Why not just do component level testing then? While this would cover most of the scenarios we’re still missing one layer of testing that still needs to be tested

The table below summarizes the integration level versus the speed/cost of each method:

There’s no reason to be completely constrained by any of the methods so the answer to the question is you want to use all of them. The question then becomes what do I test with each method and what amount of testing, per method, is enough. The answer here is going to be different for everyone and it’s going to be constrained by the product and features you’re testing.

These are the general rules I follow:

Unit Testing – Sanity tests at the lowest level. Make sure the methods can handle valid and invalid data and that the basic functionality works. No real test case scenario testing here. (Note: depending on how your organization is defined this might fall solely on the feature owner rather than the testers).

Component Level Testing – The bulk the work is here; test all the define test case scenarios here since it’s fast and gives a good amount of coverage.

UI Testing –This covers the things you can’t test through component level testing such as verifying that actual UI elements are displayed correctly. In addition to that end to end scenarios, this means running a very small subset of the scenarios testing through the component level to verify that the feature as a whole, as the user will be experiencing it, will work.

With that mix you can have full coverage of the whole feature with a major emphasis on correctness with an extra layer of testing for the UI integration. The huge advantage here is that since your correctness testing is relatively fast you can run and investigate a full test run in a fraction of the time that it would take to do a full UI test pass but without actually neglecting the UI scenarios.

One thing here that I want to make some emphasis on is that people have a preference for one type of testing over the other (which is natural) however some people end up with tunnel vision and dismiss the other methods which as we’ll see from the next example can either hurt efficiency or coverage (or both).

Case Study

Let’s take those 3 concepts on more concrete example, a simplified IntelliSense scenario.  When a call is invoked the compiler calculates the raw result data and passes it to the IDE which does some formatting and then that result is passed to the Editor to be displayed. Our three components here are the compiler, the IDE and the editor.

IntelliSense Action

Compiler

IDE

Editor

  

Now let’s go in a deeper layer and define an AutoComplete scenario with these methods defined on the components mentioned above:

·         string[] Compiler.CompleteStatement(string file, int position)

·         bool IDE.CompleteStatement(int position)

·         bool Editor.DisplayCompletions(string[] completionList)

·         bool Editor.InsertText(string text)

The action workflow would be this:

1)      IDE.CompleteStatement is called

2)      Compiler.CompleteStatement is called

3)      If there’s any results

a.       If there’s just one result call Editor.InsertText

b.      If there’s multiple results call Editor.DisplayCompletions

Finally let’s define a simple test case in which we’re invoking autocomplete on line 3, column 5 of a.cpp and I'm expecting have a list displayed with 3 results. With the specs for our action and our test case defined here’s testing divided on those 3 areas:

Unit Testing (verify that the methods can work independently, this also doesn’t take into account our scenario):

-          Verify that each of the 4 methods return the expected results given specific input

o   Compiler.CompleteStatement return results with the raw data

o   IDE.CompleteStatement invoke InsertText when there’s just one result and DisplayCompletions when there’s more than one

o   IDE.CompleteStatement formats the raw data

o   Editor.InsertText inserts the given text

o   Editor.DisplayCompletions invokes the event to display the list and passes it the relevant data? (ie which data to display, tooltip data, etc)

-          Verify that the methods don’t fail for valid inputs

-          Verify that the methods can handle invalid data gracefully

-          Verify negative cases (ie: the results should be empty)

Component Testing (at the main layer through test hooks verify the correctness of the results):

-          The IDE gets the 3 results from the compiler and formats that data correctly before passing it to the editor

UI Testing (verify that the action as a whole is successful)

-          After invoking the action in the same way the user would does the list displays with the 3 expected items.

-          Verify that it works for every way to invoke the action when there’s multiple ways defined (menus, context menu, mouse, keyboard shortcuts).

One thing you might be wondering here is wherever you would want to add component tests at the other two layers (the compiler and the editor). In this example it doesn’t makes much sense since the IDE is calling the Compiler at the component level anyway (not to mention that most likely there’s a full test suite dedicated to those scenarios at the compiler level already anyway). As for the editor it just consumes whatever the IDE feeds it so the UI tests would cover that integration.

The final result of all this is that for a more realistic test plan in which you had more than just one scenario you would only have to add tests at the component level as testing those scenarios either with unit tests or UI tests wouldn’t give you any extra coverage.

On Visual Studio 2010 we completely changed the IntelliSense architecture (more details on that from this blog post) and the problems that made people want to turn IntelliSense off are a thing of the past. Without going into much detail here, the big part of the change was in the way we did the parsing (more info on that from this blog post) and that as a result of that we’re not using NCB files anymore to store the IntelliSense information and we’re using SQLCE SDF databases to do so (and in turn retrieve the information using SQL queries). And just as a side note here, this means that if you’re developing addins for Visual Studio you pretty much have access to the same information that Visual Studio itself does regarding IntelliSense (but I’ll leave that conversation, and all the fun stuff you can do with that for another day).

 Even after those changes, we decided to provide our customers with a rich set of options to control how much of IntelliSense they want (if they want it at all). On this post I’ll go over the available option as well as some background as to what’s going on behind the scenes.

Lets start with the basics, the options are available under the Tools Menu under the Options items. This will bring up the Visual Studio options dialog. On the right side panel select Text Editor, C/C++, Advanced.

The window is fairly simple, the left pane is the shared Tools -> Options tree menu. On the right are the specific items we’re interested in, the options are divided in categories which are collapsible. Under each category (which we’ll discuss below) you can select the individual options.

There are five main categories:

·         Browsing/Navigation – These all affect the behavior of the IntelliSense database.

·         Diagnostic Logging – Enables debug logging for some of the IntelliSense features. This one is more of a troubleshooting option so I’ll skip these options.

·         Fallback Location – This controls the default behavior when there are issues with creating the IntelliSense database (for example if you’re opening a project from a network share for which you don’t have write access to).

·         IntelliSense – IntelliSense specific options

·         References – Find All References options

Now for the specifics, some of the Dialogs are self-explanatory so I won’t go into details about those.

Browsing/Navigation

Disable Database – Disable all use of the code browsing database. The database will not be opened or created.

Basically this is the main switch to turn IntelliSense off. Since no database file will be touched no IntelliSense parsing will go on. There’s another IntelliSense option (which will get to in the IntelliSense section) to just turn IntelliSense on, that still populates the database. Turning this setting on also disables everything under Browsing/Navigation and IntelliSense (except Disable #include Auto Complete).

Disable Database Updates – The code browsing database will be opened read-only and no updates will be performed.

This means, run IntelliSense with whatever was previously on my database (created from previous sessions).

Disable Database Auto Updates – The code browsing database will not be automatically updated when source files are modified.

IntelliSense information is gathered in real time, however if you got into a state in which vcpkgsvr.exe becomes unstable or where you rather not have background parsing going on you can change the setting here. On a side note, if you do get into such a state with vcpkgsvr.exe by all means open a bug on it (details on how to do so are available on this blog post).

Disable Implicit Files – The code browsing database will not collect data for files not specified in a project.

Information files that are not explicitly included in the project file (ie: external dependencies) won’t be parsed into the database so you won’t get any IntelliSense on those.

Disable Implicit Cleanup – The code browsing database will not clean up no longer referenced implicit files.

                When implicit references are removed entries related to these are removed from the database. With this option turned on that information will remain on the database.

Disable External Dependencies Folder – The ‘External Dependencies’ folder for each project will not be created/updated.

This one doesn’t affects IntelliSense related but worth talking about. All the implicit files will show up on the solution explorer as a separate node (External Dependencies). This doesn’t means that they are actually part of your project but since the IntelliSense database knows where they are and their information you get them visually represented on the Solution Explorer as if it were. Enabling this option removes that node from the tool window.

Recreate Database – Recreate the code browsing database from scratch upon the next solution load.

This is for cases where you think your database is getting corrupted and you rather start it from scratch every time you load the solution. This does means that you’ll have to wait until the solution is completely parsed until you can start getting accurate IntelliSense.

Rescan Solution Interval – A ‘Rescan Solution Now’ job will be scheduled every ‘value’ minutes. The value must be between 0 and 5000.

Changes you make are parsed in real time (ie: as you make them), however every set amount of time a complete parse is triggered to ensure that the whole database is up to date. This extra parse won’t actually clean up your database and recreate it from scratch but rather scan for changes made on files that are not active (take for example opening one of the header files your project is referencing on a different instance of Visual Studio). By default this is every 60 minutes, by changing this you can control that interval.

Fallback Location

Always Use Fallback Location – Indicates that the browsing database and IntelliSense files should always be stored in your ‘Fallback Location’, not next to the .sln file.

As the description mentions, the default behavior has always been to save the IntelliSense files on the same folder as the solution file. If you rather have them somewhere else (more specifically on the fallback location) you can change this flag.

Do Not Warn If Fallback Location Used – Do not inform or prompt you if a ‘Fallback Location’ is used.

As we mentioned before if there’s any issues with creating the SDF file a fallback location will be used. By default you get a prompt when this happens and Visual Studio will try to determine a location for you (if you actually want one). Turning on this setting will prevent that initial dialog from showing up (this is the same as selecting the “Do not prompt me again” checkbox on the dialog).

Fallback Location – This value is used as a secondary location to store the browsing database or IntelliSense files. If empty this will default to your temporary directory.

IntelliSense

Before going on with the IntelliSense settings, I’ll briefly go over the features these settings affect since most of them are new in Visual Studio 2010.

QuickInfo – This is the tooltip you get when hovering over a symbol (or using the keyboard shortcut Ctrl +K, Ctrl + I).

Squiggles – These are the red curvy lines you get on the editor window when there are errors.

Error Reporting – The errors that cause the IntelliSense squiggles to appear are also displayed on the Error Window (which you can find under the View menu or through its keyboard shortcut Ctrl + \, Ctrl + E)

More information about Squiggles and error reporting is available on this blog post.

#include Auto Complete – When including files on code you’ll get the same behavior as previous versions of Member List and AutoComplete on these statements.

More information about this new feature is available on this blog post.

Auto Quick Info – Automatically display Quick Info tooltips when hovering over text in the editor.

Turning this feature off removes the hovering (you can still get Quick Info via the keyboard shortcut).

Disable IntelliSense – Disable all IntelliSense features

As mentioned previously this will also change the values of all the rest of the options (excluding Disable #include Auto Complete)

Disable Auto Updating – IntelliSense updating will be delayed until an actual request for IntelliSense

Instead of database updates happening on real time they will happen upon request.

Disable Error Reporting – Disables reporting of IntelliSense errors through squiggles and the Error List window. Also disables the background parsing associated with error reporting.

Changing this to true completely disables both squiggles and error reporting.

Disable Squiggles – Disable IntelliSense error Squiggles.

This disables only the visible squiggles (errors still get reported through the error reporting window).

Disable #include Auto Complete – Disable automatic completion of #include statements.

Max Cached Translation Units – The maximum number of translation units that will be kept active at a time for IntelliSense requests. The value must be between 2 and 15.

A translation unit (TU) is the term for a source file plus all of its included headers. For each translation unit there’s an instance of vcpkgsvr.exe running. The tradeoff here as usual is memory versus performance, if you like working with multiple source files open in the editor (and you actually swap to them constantly) you will see some performance increase by raising this number. The default value for this is two (which is more than enough for most projects).

References

The “Find All References” was also improved for this release. Both are basically allowing you to turn the new behavior on or off (more information on the new functionality and the two options on this blog post)

Disable Resolving – ‘Find All References’ will display raw textual results instead of using IntelliSense to verify each candidate (faster).

Hide Unconfirmed – Hide unconfirmed items in the ‘Find All References’ results.

                I though ok, this should be easy enough, just call Assembly.Unload right? Well to my surprise there’s no such method. After doing some research I found there’s an unload method if you’re using AppDomain and I got all excited. Unfortunately that didn’t last long as I had to use AppDomain.Load and due to the path issue I mentioned above Load would just fail. The loading of the assembly was being done though Assembly.LoadFrom to get around the path issue.

The whole process of trial an error lasted a few hours and I probably lost a few hairs but experimenting with the Assembly.Load overloads I saw one that took a byte array and some light bulbs started flashing above my head. Using that overload solved two issues:

-          I could load the assembly no matter the path because I was getting it in memory first (inside the byte array)

-          Since I was loading it in memory and loading that memory buffer the file wasn’t getting locked so that fixed my original issue.

String filename = …; // Full path of the assembly I want to load

byte[] file = null;

int bufferSize = 1024;

using (FileStream fileStream = File.Open(filename, FileMode.Open))

{

using (MemoryStream memoryStream = new MemoryStream())

       {

              byte[] buffer = new byte[bufferSize];

              int readBytesCount = 0;

              while ((readBytesCount = fileStream.Read(buffer, 0, bufferSize)) > 0)

                     memoryStream.Write(buffer, 0, readBytesCount);

              file = memoryStream.ToArray();

       }

}

Assembly assembly = Assembly.Load(file);