Share via

Testing a Team Services extension and tracking code coverage

  • Article

Jasmine is a framework that allows you to write tests in JavaScript for AngularJs or NodeJs applications. In our case, we will use Jasmine to write tests for VSTS extension.

Implementing Jasmine on an Extension Project

The Jasmine Framework is a Jasmine.js library. To retrieve it, we have two possibilities, either with a package Nuget or with an npm package.
In case the extension is developed with Visual Studio, we retrieve the Chutzpah package with Nuget that contains jasmine and its test adapter.

image

With Visual Studio Code (or with another IDE), we can retrieve this framework with npm with the command.

npm install --save-dev jasmine

If you are using Josh Garverick’s Visual Studio Project Templates, all components required to run Jasmine tests are already included in the npm packages configured in the "packages.json" file.

image

Write your unit tests for your extension

Writing a Jasmine unit test is fairly simple. The Project Template has an example in the test folder in the "testSpec.js" file.

Here is another example:

image

Run your tests in Visual Studio or Visual Studio Code

In Visual Studio

There are several ways to run Jasmine unit tests. With Visual Studio, you can use a test adapter like Chutzpah, which is one of the best known.  It is installed with Nuget by retrieving the latest version of the Chutzpah package.

Next, to be able to visualize and execute the tests in the Visual Studio Test Explorer, install an extension to Visual Studio called Chutzpah Test Adapter for the Test Explorer. Additionally, install the Chutzpah Test Runner Context Menu Extension to be able to execute tests from the contextual menu.

image

Once installed, the tests appear in the explorer test, and they can be run as unit tests.

image

In Visual Studio Code

You can use the grunt-contrib-jasmine grunt already existing in the template, and in the "gruntfile.js" file have the jasmine task.

clip_image012

Run grunt jasmine

clip_image014

Chutzpah file configuration : Chutzpah.json

To be able to set Chutzpah parameters, we add a configuration file json (at the root of the project) named "Chutzpah.json". Inside, we specify the file of specs to be executed, the mode of compilation of the typescript files, and the parameters of execution of the code coverage.

clip_image016
For all details on this file, see https://github.com/mmanela/chutzpah/wiki/Chutzpah.json-Settings-File

Running tests as part of Continuous Integration (CI)

IMPORTANT - The following section contains paths as part of the configuration. These are solution-specific and likely to differ in your environment. Use the screenshots and examples as reference only! 

To run the JavaScript tests in a build of Continuous Integration in VSTS, add the Test Assemblies task to your build definition.

SNAGHTMLc714e3b

Set parameters

  1. Execution Options - The tests path and pattern.
  2. Advanced Execution Options - The root of test adapter : the packages folder where the Nuggets packages, including the Chutzpah package, are downloaded.

After the build, the results of the tests are obtained in the Summary tab.

image

And the details on the tests tab.

image

Code coverage analyse

Enabled the code coverage

The code coverage indicators provide an indication (in percentage) of the number of lines covered by the tests.

One of the advantages of Chutzpah is that it allows this analysis and generates a report during the tests execution.
To activate it, add the parameters in the configuration file of chutzpah.

clip_image022

  1. Enable the code coverage analysis and the files to includes and excludes.
  2. Specify the file type for the analysis result; in our case, we will choose the lcov format.

Executing the code coverage analysis during CI build

With this configuration, the code coverage analysis will be done automatically at the same time that the tests will be executed, so there is nothing more to configure in the build.

After the build runs, the result of the analysis is visible in the summary tab.

image

Publish and view the detailed analysis report

Having this percentage is a good indication, but it's better to have a detailed report that indicates on each file the lines that are not covered by the tests in order to improve them.

In the result of the VSTS builds, the code coverage tab allows you to display this report, which is in the Jacoco or Cobertura format.

The problem we have is that the report generated by Chutzpah is in lcov format, so in the build execution convert this report into a report summary of type Cobertura, then, in html format, and, finally, publish this report.

  • The first step is to convert the file "lcov.dat" obtained during the execution of the tests in "Cobertura.xml" format. This conversion is done with a Python script that is available here. Add this script into the repository, and execute it during the build with a Command Line task.
    SNAGHTMLc72ef1a

Example:

$(Build.SourcesDirectory)\CountdownWidget\lcov_cobertura.py $(Build.SourcesDirectory)\CountdownWidget\CountdownWidget\lcov.dat --output $(Build.BinariesDirectory)\Cobertura.xml

  • It is necessary to generate a report in html format from the "Cobertura.xml" using the tool ReportGenerator which is obtained in a package Nuget (included in our project), available here.
    SNAGHTMLc746cf0

Examples:

$(Build.SourcesDirectory)/CountdownWidget/packages/ReportGenerator.2.5.1/tools/ReportGenerator.exe

-reports:$(Build.BinariesDirectory)\Cobertura.xml -targetdir:$(Build.BinariesDirectory)\reports -reporttypes:Html;Xml;XmlSummary

  • The last step is to publish the report generated by ReportGenerator by using the Publish Code coverage result task.
    SNAGHTMLc75d337

Examples:

$(Build.BinariesDirectory)/Cobertura.xml

$(Build.BinariesDirectory)/reports

Once the build is running, you get the report in the Code Coverage tab.

image

 

Checklist

  1. Add Jasmine to your project
  2. Add and validate unit tests
  3. Add and configure Test Assemblies task to your CI build
  4. Configure Chutzpah configuration to enable code coverage
  5. Publish code coverage results

Comments

  • Anonymous
    February 25, 2017
    There are easier ways to publish the code coverage report, since Chutzpah can also output in JaCoCo format and an HTML report, using the Transforms settings (it can output multiple formats at once). See here https://medium.com/nov-rsis-development/enabling-javascript-typescript-code-coverage-in-a-vsts-tfs-build-36fe41d40f97#.qe6ly49kq
    • Anonymous
      February 28, 2017
      Chutzpa provide xml summary JaCoCo code coverage but doesn't provide the Html report Jacoco or Cobertura format for the Code coverage tab in the build, so I need to use Report Generator for tranform xml summary to html compatible report
  • Anonymous
    March 09, 2018
    Any hints on using this with typescript (and webpack)? I tried addingimport {} from 'jasmine';to the top of your example spec for adding two numbers, but when I do npm run test it says 'define is not defined':C:\git\myproject>npm test> myproject@0.0.1 test C:\git\myproject> jasmine-ts "src/spec/**/*.spec.ts"C:\git\myproject\src\spec\test.spec.ts:1import {} from 'jasmine'; ^ReferenceError: define is not defined at Object. (C:\git\vsts-team-calendar\src\spec\test.spec.ts:1:63) at Module._compile (module.js:635:30) at Module.m._compile (C:\git\myproject\node_modules\ts-node\src\index.ts:392:23) at Module._extensions..js (module.js:646:10) at Object.require.extensions.(anonymous function) [as .ts] (C:\git\myproject\node_modules\ts-node\src\index.ts:395:12) at Module.load (module.js:554:32) at tryModuleLoad (module.js:497:12) at Function.Module._load (module.js:489:3) at Module.require (module.js:579:17) at require (internal/module.js:11:18)npm ERR! Test failed. See above for more details.C:\git\myproject>I'm also using webpack, not sure if I have to do something with that?
  • Anonymous
    April 27, 2018
    Under the heading "In Visual Studio Code", you say: "You can use the grunt-contrib-jasmine grunt already existing in the template, and in the "gruntfile.js" file have the jasmine task."What project template? The only project template I see is the Visual Studio template you mentioned, but this isn't Visual Studio, it's VS Code.