Testing tools for Visual Studio extensions

Question

Testing tools for Visual Studio extensions

Shai Geva 26

Hi,
I'm developing a new extension for Visual Studio and looking for how to test it (except for standard unit tests, of course).

I was hoping for a testing infrastructure that can run a light-weight UI-less Visual Studio instance, but couldn't find anything in the docs.
Does something like this exist? If not, are there any recommended testing practices to follow?

Thank you!

Accepted answer

1 additional answer

Your answer

Answer 1

Sergey Vlasov 266

VsixTesting allows you to easily test your Visual Studio Extensions: https://github.com/josetr/VsixTesting

Shai Geva 26 Reputation points

2021-08-19T13:14:00.9+00:00

Will dig into it.
From initial glance, it looks cool, although not very extensive (I specifically need capabilities around content of files and auto-completion, and these don't appear to be covered).
But it appears to be the closest thing out there.

Thank you!

Answer 2

Michael Taylor 60,326

To be honest I think it completely depends upon your extension. Assuming you have unit tests that are mocking the various states that your extension relies on then the only other testing that would probably make sense is actually integration testing with VS itself. Everything else is just going to give you potentially false positives.

As an example, your extension may request a service from VS. That call will likely work just fine in any test harness you have but fail in VS itself because: a) VS is calling your extension on a non-UI thread and the service isn't available there, or b) the service isn't available yet because the solution isn't loaded. These types of situations would be very hard to know about in advance and therefore test against. It is these types of situations that you would only know by running VS (and often only in certain circumstances). But then you'll detect an issue and add a unit test for it.

So I believe the correct approach is to simply run integration tests using either the experimental instance of VS or perhaps a named instance that is installed on your test machines.

I would also recommend that you ensure your extension is running the VS extension analyzer that will help detect these kinds of problems at build time as well but you might already be doing that.

Shai Geva 26 Reputation points

2021-08-17T14:15:32.667+00:00

Thank you for the super-quick answer!
I’m aware of the option to mock out everything and would do it if that’s the best option.
(I'm writing a lot of standard UTs for business logic itself, this is indeed more about testing the integration with VS, which is a big part of the extension)

The issue, of course, is that my mocks would need to closely resemble what Visual Studio does in practice, and that’s a challenge.

I would prefer, if possible, to also have (automated) integration tests where Visual Studio (mostly) runs real instances of services, as this tends to be closer to reality (similar to the utilities on the Intellij platform: https://plugins.jetbrains.com/docs/intellij/light-and-heavy-tests.html).

I understand from your answer that such an infrastructure does not exist?

Also - you suggested using actual VS instances to run integration tests - were you talking about automated testing or manual testing? Is there a way to automate actions on actual instances? For example, to programmatically tell a VS instance “switch to that file at that location, try to auto-complete”?

Thank you!
Michael Taylor 60,326 Reputation points

2021-08-17T14:43:45.477+00:00

You can use a UI testing tool like Appium to test any Windows app, including VS. You'll have to read up on it though as I don't use it myself. Nevertheless you should be able to create integration tests that start the experimental (or whatever) instance of VS on the test machine and then navigate through it like you would everywhere else. I'm not aware of an existing, publicly accessible set of UI tests for VS but I suspect MS may have something internal. Nevertheless it'll be slow but the most accurate way to test your extension.

As for getting some UI stuff automated like auto-complete, that could be tricky. You'll need to read up on the testing framework you decide to use to see how to accomplish that. It is more a question related to your testing tool than what you're testing to me.

Share via

Testing tools for Visual Studio extensions

1 additional answer

Your answer