Share via

A little about our UI-less test harness

  • Article

A question that comes up quite frequently when we introduce people to OneNote automation concerns our UI-less automation framework.

I do not like UI based automation.  I know there are many automation systems in the market that drive UI based automation (clicking buttons and the like) and even within Office we have several tools that are maintained by us to take similar actions.  I understand the benefit these tools provide in that they more accurately allow automation to mimic real user behavior.  But in an automation run, they always seem unreliable to me - there may be a window that opens and steals focus, some bug with the tool itself causes a Windows message to be lost.  With thousands of tests being run each day, every day, you can imagine even an intermittent bug with the automation system causing "noisy" failures that distract from reliable automation.  Additionally, there is a time element these tools need - the entire UI needs to be rendered before the test can be run.  If the goal of the test is to verify some element that does not need the UI worked properly (file IO and/or sync are examples), the UI is not even needed.

So what we do in OneNote is roughly this:

  1. Start OneNote
  2. Load the onmain.dll into memory
  3. Load our test harness
    1. Our test harness uses .NET reflection to load the onmain.dll and reference the actions in it.  This is a vast simplification, but should do for this level of discussion.
    2. Now all the actions within the onmain.dll are exposed to our test harness as well.
  4. Finally, we can invoke any action we want from our test harness.

In other words, at this point, if I want to mimic the user clicking the "Bold" button to make text bold, I don't need the bold button to be visible.  I can just invoke the click event that would have fired and have that code run immediately. 

There is a downside here.  First, any UI testing that might have been covered is lost.  For instance, suppose there was a bug in that the bold button is never enabled.  My automation will not pick up that error.  Second, I have to be careful with context.  Although I can invoke the code that runs when a user would have clicked Bold, I need to make sure I have a page with a cursor on it before doing so. 

But since we went to this system our automation has been much more reliable and once testers learn the system, we get much better coverage.

Questions, comments, concerns and criticisms always welcome,
John

Comments

  • Anonymous
    September 20, 2013
    How are the pass / fail results reported ?   How does the non-UI test harness exercise mouse drag behavior like that cited here ?   answers.microsoft.com/.../a7a3aa51-eaf8-4dd4-94f4-7c8351b76f76  

  • Anonymous
    September 22, 2013
    We perform verifications for tests like this using an object model based on our xml - very similar to the OM on codeplex.