July 2013
Volume 28 Number 7
Data Points - Behavior-Driven Design with SpecFlow
By Julie Lerman
.jpg)
By now you’re familiar with my penchant to invite developers to give talks on topics I’ve been curious about at the user group I lead in Vermont. This has resulted in columns on topics such as Knockout.js and Breeze.js. There are still more topics, such as Command Query Responsibility Segregation (CQRS), that I’ve been chewing on for a while. But recently Dennis Doire, an architect and tester, spoke on SpecFlow and Selenium, two tools for testers doing behavior-driven development (BDD). Once again, my eyes got wide and my mind started looking for excuses to play with the tools. Really, though, it was BDD that got my attention. Even though I’m a data-driven person, my days of designing applications from the database up are far behind me and I’ve become interested in focusing on the domain.
BDD is a twist on test-driven development (TDD) that focuses on user stories and building up logic and tests around those stories. Rather than satisfying a single rule, you satisfy sets of activities. It’s very holistic, which I love, so this perspective interests me a great deal. The idea is that while a typical unit test might ensure that a single event on a customer object works properly, BDD focuses on the broader story of the behavior that I, the user, expect when I’m using the system you’re building for me. BDD is often used to define acceptance criteria during discussions with clients. For example, when I sit in front of the computer and fill out a New Customer form and then hit the Save button, the system should store the customer information and then show me a message that the customer has been successfully stored.
Or perhaps when I activate the Customer Management portion of the software, it should automatically open up the most recent Customer I worked on in my last session.
You can see from these user stories that BDD might be a UI-oriented technique for designing automated tests, but many of the scenarios are written before a UI has been designed. And thanks to tools such as Selenium (docs.seleniumhq.org) and WatiN (watin.org), it’s possible to automate tests in the browser. But BDD isn’t just about describing user interaction. To get a view of the bigger picture of BDD, check out the panel discussion on InfoQ among some of the authorities on BDD, TDD and Specification by Example at bit.ly/10jp6ve.
I want to step away from worrying about button clicks and such and redefine the user stories a little. I can remove the UI-dependent elements of the story and focus on the part of the process that isn’t dependent on the screen. And, of course, the stories I’m interested in are the ones related to data access.
Building up the logic to test that a particular behavior is being satisfied can be tedious. One of the tools Doire demonstrated in his presentation was SpecFlow (specflow.org). This tool integrates with Visual Studio and enables you to define user stories—called scenarios—using its simple rules. It then automates some of the creation and execution of the methods (some with tests and some without). The goal is to validate that the rules of the story are being satisfied.
I’m going to walk you through creating a few behaviors to whet your appetite, and then if you’re looking for more, you’ll find some resources at the end of the article.
First you need to install SpecFlow into Visual Studio, which you can do from the Visual Studio Extensions and Updates Manager. Because the point of BDD is to begin your project development by describing behaviors, the first project in your solution is a test project in which you’ll describe these behaviors. The rest of the solution will flow from that point.
Create a new project using the Unit Test Project template. Your project will need a reference to TechTalk.SpecFlow.dll, which you can install using NuGet. Then create a folder called Features inside this project.
My first feature will be based on a user story about adding a new customer, so inside the Features folder, I create another called Add (see Figure 1). Here’s where I’ll define my scenario and ask SpecFlow to help me out.
.png "Test Project with Features and Add Sub-Folders")
Figure 1 Test Project with Features and Add Sub-Folders
SpecFlow follows a specific pattern that relies on keywords that help describe the feature whose behavior you’re defining. The keywords come from a language called Gherkin (yes, as in pickle), and this all originates from a tool called Cucumber (cukes.info). Some of these keywords are Given, And, When, and Then, and you can use them to build a scenario. For example, here’s a simple scenario, which is encapsulated in a feature—Adding a New Customer:
Given a user has entered information about a customer
When she completes entering more information
Then that customer should be stored in the system
You could be more elaborate, for example:
Given a user has entered information about a customer
And she has provided a first name and a last name as required
When she completes entering more information
Then that customer should be stored in the system
That last statement is where I’ll be doing some data persistence. SpecFlow doesn’t care about how any of this happens. The goal is to write scenarios to prove the outcome is and remains successful. The scenario will drive the set of tests and the tests will help you flesh out your domain logic:
Given that you have used the proper keywords
When you trigger SpecFlow
Then a set of steps will be generated for you to populate with code
And a class file will be generated that will automate the execution of these steps on your behalf
Let’s see how this works.
Right-click the Add folder to add a new item. If you’ve installed SpecFlow, you can find three SpecFlow-related items by searching on specflow. Select the SpecFlow Feature File item and give it a name. I’ve called mine AddCustomer.feature.
A feature file starts out with a sample—the ubiquitous math feature. Notice that the Feature is described at the top, and on the bottom a Scenario (which represents a key example of the feature) is described using Given, And, When and Then. The SpecFlow add-in ensures that the text is color-coded so you can easily discern the step terms from your own statements.
I’ll replace the canned feature and steps with my own:
Feature: Add Customer
Allow users to create and store new customers
As long as the new customers have a first and last name
Scenario: HappyPath
Given a user has entered information about a customer
And she has provided a first name and a last name as required
When she completes entering more information
Then that customer should be stored in the system
(Thanks to David Starr for the Scenario name! I stole it from his Pluralsight video.)
What if the required data isn’t provided? I’ll create another scenario in this feature to handle that possibility:
Scenario: Missing Required Data
Given a user has entered information about a customer
And she has not provided the first name and last name
When she completes entering more information
Then that user will be notified about the missing data
And the customer will not be stored into the system
That will do for now.
From User Story to Some Code
So far you’ve seen the Feature item and the color-coding that SpecFlow provides. Note that there’s a codebehind file attached to the feature file, which has some empty tests that were created based on the features. Each of those tests will execute the steps in your scenario, but you do need to create those steps. There are a few ways to do that. You could run the tests, and SpecFlow will return the code listing for the Steps class in the test output for you to copy and paste. Alternatively, you could use a tool in the feature file’s context menu. I’ll describe the second approach:
- Right-click in the text editor window of the feature file. On the context menu, you’ll see a section dedicated to SpecFlow tasks.
- Click Generate Step Definitions. A window will pop up verifying the steps to create.
- Click the Copy methods to clipboard button and use the defaults.
- In the AddCustomer folder in your project, create a new class file named Steps.cs.
- Open the file and, inside the class definition, paste the clipboard contents.
- Add a namespace reference to the top of the file, using TechTalk.SpecFlow.
- Add a Binding annotation to the class.
The new class is listed in Figure 2.
Figure 2 The Steps.cs File
[Binding]
public class Steps
{
[Given(@"a user has entered information about a customer")]
public void GivenAUserHasEnteredInformationAboutACustomer()
{
ScenarioContext.Current.Pending();
}
[Given(@"she has provided a first name and a last name as required")]
public void GivenSheHasProvidedAFirstNameAndALastNameAsRequired
()
{
ScenarioContext.Current.Pending();
}
[When(@"she completes entering more information")]
public void WhenSheCompletesEnteringMoreInformation()
{
ScenarioContext.Current.Pending();
}
[Then(@"that customer should be stored in the system")]
public void ThenThatCustomerShouldBeStoredInTheSystem()
{
ScenarioContext.Current.Pending();
}
[Given(@"she has not provided both the firstname and lastname")]
public void GivenSheHasNotProvidedBothTheFirstnameAndLastname()
{
ScenarioContext.Current.Pending();
}
[Then(@"that user will get a message")]
public void ThenThatUserWillGetAMessage()
{
ScenarioContext.Current.Pending();
}
[Then(@"the customer will not be stored into the system")]
public void ThenTheCustomerWillNotBeStoredIntoTheSystem()
{
ScenarioContext.Current.Pending();
}
}
If you look at the two scenarios I’ve created, you’ll notice that while there’s some overlap in what’s defined (such as “a user has entered information about a customer”), the generated methods do not create duplicate steps. It’s also notable that SpecFlow will leverage the constants in the method attributes. The actual method names are irrelevant.
At this point, you could let SpecFlow run its tests that will call these methods. While SpecFlow supports a number of unit-testing frameworks, I’m using MSTest, so if you’re looking at this solution in Visual Studio, you’ll see that Feature’s codebehind file defines a TestMethod for each scenario. Each TestMethod executes the correct combination of step methods with a TestMethod that’s run for the HappyPath scenario.
If I were to run this now, by right-clicking the Feature file and choosing “Run SpecFlow Scenarios,” the test would be inconclusive, with the message: “One or more step definitions are not implemented yet.” That’s because each of the methods in the Steps file are all still calling Scenario.Current.Pending.
So, it’s time to flesh out the methods. My scenarios tell me I’ll need a Customer type with some required data. Thanks to other documentation, I know that the first and last name are currently required, so I’ll need those two properties in the Customer type. I also need a mechanism for storing that customer, as well as a place to store it. My tests don’t care how or where it’s stored, just that it is, so I’ll use a repository that will be responsible for getting and storing data.
I’ll start by adding _customer and _repository variables to my Steps class:
private Customer _customer;
private Repository _repository;
And then stub out a Customer class:
public class Customer
{
public int Id { get; set; }
public string FirstName { get; set; }
public string LastName { get; set; }
}
That’s enough to let me add code to my step methods. Figure 3 shows the logic added to the steps related to HappyPath. I create a new customer in one, then I provide the required first and last names in the next. There’s not really anything I need to do to elaborate on the WhenSheCompletesEnteringMoreInformation step.
Figure 3 Some of the SpecFlow Step Methods
[Given(@"a user has entered information about a customer")]
public void GivenAUserHasEnteredInformationAboutACustomer()
{
_newCustomer = new Customer();
}
[Given(@"she has provided a first name and a last name as required")]
public void GivenSheHasProvidedTheRequiredData()
{
_newCustomer.FirstName = "Julie";
_newCustomer.LastName = "Lerman";
}
[When(@"she completes entering more information")]
public void WhenSheCompletesEnteringMoreInformation()
{
}
The last step is the most interesting. This is where I not only store the customer, but prove that it has indeed been stored. I’ll need an Add method in my repository to store the customer, a Save to push it into the database and then a way to see if that customer can, in fact, be found by the repository. So I’ll add an Add method, a Save and a FindById method to my repository, like so:
public class CustomerRepository
{
public void Add(Customer customer)
{ throw new NotImplementedException(); }
public int Save()
{ throw new NotImplementedException(); }
public Customer FindById(int id)
{ throw new NotImplementedException(); }
}
Now I can add logic to the final step that will be called by my HappyPath scenario. I’ll add the customer to the repository and test to see if it can be found in the repository. This is where I finally use an assertion to determine if my scenario is succeeding. If a customer is found (that is, IsNotNull), the test passes. This is a very common pattern for testing that data has been stored. However, from my experience with Entity Framework, I see a problem that won’t get caught by the test. I’ll start with the following code so I can show you the problem in a way that may be more memorable than just showing you the correct way to begin with (I’m mean that way):
[Then(@"that customer should be stored in the system")]
public void ThenThatCustomerShouldBeStoredInTheSystem()
{
_repository = new CustomerRepository();
_repository.Add(_newCustomer);
_repository.Save();
Assert.IsNotNull(_repository.FindById(_newCustomer.Id));
}
When I run my HappyPath test again, the test fails. You can see in Figure 4 that the test output shows how my SpecFlow scenario is working out so far. But pay attention to the reason the test failed: It’s not because the FindById didn’t find the customer, it’s because my repository methods are not yet implemented.
Figure 4 Output from Failed Test Showing Status of Each Step
Test Name: HappyPath
Test Outcome: Failed
Result Message:
Test method UnitTestProject1.UserStories.Add.AddCustomerFeature.HappyPath threw exception:
System.NotImplementedException: The method or operation is not implemented.
Result StandardOutput:
Given a user has entered information about a customer
-> done: Steps.GivenAUserHasEnteredInformationAboutACustomer() (0.0s)
And she has provided a first name and a last name as required
-> done: Steps. GivenSheHasProvidedAFirstNameAndALastNameAsRequired() (0.0s)
When she completes entering more information
-> done: Steps.WhenSheCompletesEnteringMoreInformation() (0.0s)
Then that customer should be stored in the system
-> error: The method or operation is not implemented.
So, my next step is to provide logic to my repository. Eventually I’ll use this repository to interact with the database and, as I happen to be a fan of Entity Framework, I’ll use an Entity Framework DbContext in my repository. I’ll start by creating a DbContext class that exposes a Customers DbSet:
public class CustomerContext:DbContext
{
public DbSet<Customer> Customers { get; set; }
}
Then I can refactor my CustomerRepository to use the CustomerContext for persistence. For this demo, I’ll work directly against the context rather than worrying about abstractions. Here’s the updated CustomerRepository:
public class CustomerRepository
{
private CustomerContext _context = new CustomerContext();
public void Add(Customer customer
{ _context.Customers.Add(customer); }
public int Save()
{ return _context.SaveChanges(); }
public Customer FindById(int id)
{ return _context.Customers.Find(id); }
}
Now when I rerun the HappyPath test, it passes and all of my steps are marked as done. But I’m still not happy.
Make Sure Those Integration Tests Understand EF Behavior
Why am I not happy when my tests pass and I see the pretty green circle? Because I know that the test is not truly proving that the customer was stored.
In the ThenThatCustomerShouldBeStoredInTheSystem method, comment out the call to Save and run the test again. It still passes. And I didn’t even save the customer into the database! Now do you smell that funny odor? It’s the smell of what’s known as a “false positive.”
The problem is that the DbSet Find method that I’m using in my repository is a special method in Entity Framework that first checks the in-memory objects being tracked by the context before going to the database. When I called Add, I made the CustomerContext aware of that customer instance. The call to Customers.Find discovered that instance and skipped a wasted trip to the database. In fact, the ID of the customer is still 0 because it hasn’t been stored yet.
So, because I’m using Entity Framework (and you should take into account the behavior of any object-relational mapping [ORM] framework you’re using), I have a simpler way to test to see if the customer truly got into the database. When the EF SaveChanges instruction inserts the customer into the database, it will pull back the new database-generated ID of the customer and apply it to the instance that it inserted. Therefore, if the new customer’s ID is no longer 0, I know my customer really did get into the database. I don’t have to re-query the database.
I’ll revise the Assert for that method accordingly. Here’s the method I know will do a proper test:
[Then(@"that customer should be stored in the system")]
public void ThenThatCustomerShouldBeStoredInTheSystem()
{
_repository = new CustomerRepository();
_repository.Add(_newCustomer);
_repository.Save();
Assert.IsNotNull(_newCustomer.Id>0);
}
It passes, and I know it’s passing for the right reasons. It’s not uncommon to define a failing test, for example, using Assert.IsNull(FindById(customer.Id) to make sure you’re not passing for the wrong reason. But in this case, the problem still wouldn’t have shown up until I removed the call to Save. If you’re not confident about how EF works, it would be wise to also create some specific integration tests, unrelated to the user stories, to ensure that your repositories are behaving the way you expect.
Behavior Test or Integration Test?
As I traversed the learning curve of working out this first SpecFlow scenario, I encountered what I thought was a slippery slope. My scenario states that the customer should be stored in “the system.”
The problem is that I wasn’t confident about the definition of the system. My background tells me the database or at least some persistence mechanism is a very important part of the system.
The user doesn’t care about repositories and databases—just her application. But she won’t be very happy if she logs back into her application and can’t find that customer again because it never really got stored in the database (because I didn’t think that _repository.Save was necessary for fulfilling her scenario).
I consulted with yet another Dennis, Dennis Doomen, the author of Fluent Assertions and a heavy practitioner of BDD, TDD and more in large enterprise systems. He confirmed that, as a developer, I should certainly apply my knowledge to the steps and tests even if this means going beyond the intent of the user who defined the original scenario. Users provide their knowledge and I add mine in a way that doesn’t push my technical perspective into the user’s face. I continue to speak her language and communicate well with her.
Keep Digging into BDD and SpecFlow
I’m pretty sure that if it weren’t for all of the tools that have been built to support BDD, my path into it wouldn’t have been so easy. Even though I’m a data geek, I care very much about working with my clients, understanding their businesses and ensuring they have a happy experience using the software I help build for them. This is why Domain Driven Design and Behavior Driven Design speak so loudly to me. I think many developers feel the same way—even if it’s deep down in their gut (or heart)—and may also be inspired by these techniques.
Besides the friends who helped me get to this point, here are some of the resources I found useful. The MSDN Magazine article, “Behavior-Driven Development with SpecFlow and WatiN,” which can be found at msdn.microsoft.com/magazine/gg490346, was quite helpful. I also watched an excellent module from David Starr’s Test First Development course on Pluralsight.com. (In fact, I watched that module a number of times.) I found the Wikipedia entry on BDD (bit.ly/LCgkxf) to be interesting in that it presented the bigger picture of BDD’s history and where it fits in with other practices. And I’m eagerly awaiting the book, “BDD and Cucumber,” which Paul Rayner (who also advised me here) is coauthoring.
Julie Lerman is a Microsoft MVP, .NET mentor and consultant who lives in the hills of Vermont. You can find her presenting on data access and other Microsoft .NET topics at user groups and conferences around the world. She blogs at thedatafarm.com/blog and is the author of “Programming Entity Framework” (2010) as well as a Code First edition (2011) and a DbContext edition (2012), all from O’Reilly Media. Follow her on Twitter at twitter.com/julielerman.
Thanks to the following technical experts for reviewing this article: Dennis Doomen (Aviva Solutions) and Paul Rayner (Virtual Genius)
Dennis Doomen is Principal Consultant at Aviva Solutions (The Netherlands), occasional speaker, agile coach, author of the Coding Guidelines for C# 3.0, 4.0 and 5.0, the Fluent Assertions framework and the Silverlight Cookbook. He currently builds enterprise-class solutions based on .NET, Event Sourcing and CQRS. He loves agile development, architecture, Xtreme Programming and Domain Driven Design. You can reach him on twitter using @ddoomen.