Partager via

C# IntelliSense tests

  • Article

C# IntelliSense tests

 

Hi, I am Jeremy Meng from the C# QA team. I mainly work in testing the C# IntelliSense area. C# IntelliSense is a set of IDE features that help C# developers write their code faster and less error-prone. Examples of these features include

 

· Quick Info. The yellow tooltip window with information while hovering over an identifier

quick info

· Parameter Help. The yellow tooltip window showing the signature of the method and the parameter being typed

parameter help

· Completion List. The list box containing a set of completion candidates for developer to use

completion list

 

The goal of our tests is to ensure that the IntelliSense features work correctly as expected.

 

Good automated tests should be easy to maintain and easy to investigate when there are failures. They should run as fast as possible too. These good qualities would make our test pass runs shorter, thus enable us to do more runs to discover more potential issues.

 

In the past most of our C# IntelliSense tests use UI tests. We have libraries that enable us to navigate, validate and manipulate the UI of any Windows-based application. Take the Quick Info feature for example. If we want to validate the Quick Info text for a certain identifier in a C# source file opened in Visual Studio, we can use API’s to move the cursor to the position of the identifier, invoke the feature to display the Quick Info, then capture the text in the tooltip window. All of these can be done programmatically. An automated UI test behaves almost as same as a tester would perform the test manually. So it’s most desirable if we want our tests to be as close to the real usage case as possible. UI tests are also useful when there are no other existing frameworks to test a feature. However there are several issues for pure UI tests. They are slow. Synchronization problems (timing issues) happen frequently in and many of the issues are hard to debug too. So currently we are trying to reduce the usage of UI tests and write tests in other better ways that don’t involve many UI interactions.

 

For some parts of the automation we can use Visual Studio Extensibility. Visual Studio Extensibility is an assembly-wrapped COM library containing the objects and members for Visual Studio core. Things that Visual Studio Extensibility can do include creating solutions and projects, adding files, accessing core Visual Studio windows, executing Visual Studio commands, etc. without having to click menus and dialogs. Although Visual Studio Extensibility is non-UI, it only has VS core functionalities. To test our C# IntelliSense features in a non-UI way, we have to have more.

 

Instead of manipulating the UI to validate a feature, we can test the feature directly. To achieve this we do tests on component level. Look at the Quick Info example again. We could ask the C# IntelliSense Engine directly: What quick info would you show for the identifier at line x, column y in the editor?

We can then validate the text by either using a baseline, or some other kinds of dynamic verification logic, thus avoid going through the UI to obtain the quick info text.

 

Currently we can do component-level tests for IntelliSense features like Quick Info, Parameter Help, Completion Lists, as well as for other C# IDE features like Find All References, Go To Definition, Extract Method, Format, etc. 

 

We do component-level testing in two kinds of tests: semi-UI tests and non-UI tests. The two kinds of tests both have their pros and cons.

 

In the semi-UI test, we start an instance of VS, set up the solution and project, open the source file, then invoke the test hook and validate the feature. In such a test the feature is tested in a real C# project in the real Visual Studio.

 

Pros:

· Closer to real world scenarios.

· Availability of automation support from Visual Studio

· Covering integration of C# language service features and Visual Studio as well

 

Cons: 

· Starting and setting up Visual Studio take a lot of time.

· Not 100% focus on testing language service features

 

For non-UI tests, we have a framework called Stand-Alone-Language-Service-API (SALSA). SALSA utilizes the unit test framework that our developers are using. In a SALSA test, we don’t start a Visual Studio instance. We are using a mocked environment and other mocked services that are necessary to test the IDE language service features. Our data show that SALSA tests are at least 90% faster than the semi-UI tests.

 

Pros:

· Much faster

· Focus on features

 

Cons:

· Mocked environment not the same as the real Visual Studio

· Tests are limited by the developers' unit test framework so there are tests that can’t be written in SALSA

 

Conclusion:

 

We want to increase the percentage of non-UI tests since it’s much faster while keep enough amount of UI tests because they can cover integration and they are closer to the real usage scenarios.

Comments

  • Anonymous
    October 19, 2007
    Hi Yu,I noticed a problem in the completion list involving the use of enums. When you declare a variable of type nullable of an enum (MyEnum? variable = ). when you hit ctrl+space after the equals sign it show the completition list but the default value selected in the list is not the enum type (MyEnum) but the last used typed in the previous completition list.Luciano.
  • Anonymous
    October 26, 2007
    Hi Luciano,Thanks for reporting this issue to us!  There are two things happening here:(1) IntelliSense didn't offer a completion list for the nullable type of a Enum. This is a bug and has been known for a while. We might fix it in a future release but it's a low priority one. The result is that in MyEnum? variable =[SPACE] you don't get an auto-completion list after typing the space.(2) Because of (1), when you use Ctrl+space, a generic completion list will show up with the most recently selected item pre-selected.  This is by design.Please let me know if you have any questions/comments.  Thanks again!Yu Meng
  • Anonymous
    October 30, 2007
    I'm sure that moving away from pure UI tests will be more performant.I'm just struck by the fact that not having pure UI tests probably resulted in the Excel 2007 65536 bug.Not complaining here - but this is the first thing that came to mind!