Inter-sections.net : Category rspec, everything about rspec

Array matcher

Mon, 26 Nov 2007 20:04:00 +0000

Here’s a lovely little snippet that shows the reasons why Ruby is such a nice language to work with. It’s to do with matching at least one of a series of items in RSpec.

Step 1: First, it was simple

Then("user $email should receive an email with his confirmation link") do |email|
  mail = ActionMailer::Base.deliveries.last
  user = User.find_by_email(email)
  mail.body.match(/#{user.confirmation_code}/) && mail.body.match(/#{user.id}/)
end

This is my simple matcher that I started with… The world was simple, and this worked fine. But then, something happened. The world started sending more than a single email in one Story, and so the need for matching multiple emails was born.

Step 2: Then, it was easy, but messy

Then("user $email should receive an email with his confirmation link") do |email|
  matches = 0
  ActionMailer::Base.deliveries.each do |mail|
    user = User.find_by_email(email)
    matches += 1 if (mail.body.match(/#{user.confirmation_code}/) && mail.body.match(/#{user.id}/))
  end
  matches.should == 1
end

This works. It works well. It works fine. But, it’s ugly. It’s not readable. It’s not RSpec-like, and it stands out like a sore thumb in the middle of my step matchers. I have Ruby to thank for the fact that fugly hacks like this stand out and beg to be rewritten in a neater way. Other languages, such as PHP, do not have this quality, and so hacks remain untouched for ages and ages.

I could not let things rest in this state. Ugliness like that must be removed. Trying to match any one of an array of items is one of those things that are likely to be repeated a few times, so I decided to extract that out. This is where another one of Ruby’s qualities came forth, and made it child’s play to extract this concept and stick it where it belongs - in Enumerable (but, I hasten to add, only when running specs…).

Step 3: Then, it was simple again

Enter my latest addition to /stories/helper.rb:

ENV["RAILS_ENV"] = "test"
require File.expand_path(File.dirname(__FILE__) + "/../config/environment")
require 'spec/rails/story_adapter'

module Enumerable
  # my_array.should_match_at_least_one { |item| item.match(/abc/) && item.match(/def/) }
  def at_least_one_should
    matches = 0
    self.each do |item|
      matches +=1 if yield item
    end
    matches.should >= 1
  end
end

Which means that the step matcher can now been reduced to the reasonably elegant and expressive:

Then("user $email should receive an email with his confirmation link") do |email|
  user = User.find_by_email(email)
  ActionMailer::Base.deliveries.at_least_one_should do |mail|
    mail.body.match(/#{user.confirmation_code}/) && mail.body.match(/#{user.id}/)
  end
end

You might be tempted to point out that I could probably use String#include? instead of String#match, since I do not care about the actual matching, only about the presence… and you’d be right. But, like an imperfectly perfect persian rug, I’m happy with my code like this.

What to test and specify, and where to do it

Fri, 19 Oct 2007 12:33:00 +0000

So I’ve been struggling a bit with my specifications approach lately. Well, only until I found out about the new Story Runner functionality in rspec. That’s resolved a number of things, by providing a powerful tool to prepare integration tests (in the form of User Stories). From the look of things, it seems like many developers like myself have been lingering in the limbo of no-integration-tests-land, simply because rspec lacked the capability, Test::unit feels unclean, and the full-on rspec-with-watir-or-selenium seems just too large a step when you don’t even have any integration tests.

Apart from this obvious benefit, the new Story functionality has allowed me to give some more thought to something that’s been bothering me. Namely, “What do you test/specify”, and where do you do it. One of the salient issues linked to this is how to specify validations. Hang in there for a potential answer as we go through this article (as always, comments are most welcome).

What’s in an app?

That’s a key question when determining what to test and how. We’re looking at rails apps, here, so the basic model is MVC. So is this what’s in an app?

This is the basic model of a Rails app. However, from the perspective of deciding what to specify and test, it’s incomplete. I’m not talking about helper classes which belong in one of those three boxes. I’m not even talking about things like daemons, or mailers, or those kinds of components, because actually they still fit in the MVC model, just one without end-user interaction. What’s missing from this diagram, I believe, is the dependencies.

One of the problems which I (and many people, looking at the rspec-users mailing list and the #rspec channel on freenode) have hit many times and which is very irritating is what to do about those validators and relationships. How do I specify my polymorphic has_many :through, eh?

First, the big stuff

The first point I’d like to make in this article is that with the Story engine, we now have the tools to specify almost the entire system. The mapping is fairly simple:

First, you write Stories (in an outcome-focused way, with the “Given/When/Then” approach), which are essentially a much nicer way to express integration tests in a business-focused way.

Next, you write specification to define how your code will make those stories pass. You write a behaviour-focused specification each for the model, the view, and the controller, because these are the key architectural separations. Please note that this doesn’t imply you should write a specification for each method - specifications should be linked to behaviours, not low-level programming constructs. However, they are inevitably influenced by the architecture of the system.

Finally, once the specifications pass, the stories should pass too. If later, you refactor or change your code, you have your integration tests holding you up and making sure your system is still working as a whole.

You have two scenarios for a change: an internal change (e.g. refactoring) or an external change (change in the user requirements). The first is driven by a change in the code, the second is driven by a change in the Story.

If you change your code, the following things sequence of events should happen:

First, the spec for the item you changed will break. Start by fixing that spec to be correct.
Simultaneously, if you changed the behaviour, some of your Stories will break. This is not because they are wrong, though, only because your specs are mismatched - ie you’ve updated your spec for your item, but not for its dependencies. Don’t change your Stories at this point.
You proceed to update the specs for everything which was dependent on the item you changed. They break. You update the code. They pass. You’ll know that this process is complete because your Stories will pass again.

If you change the Stories, the process is subtly different:

If you change the Story, it will start by breaking.
You then change some specs to make the Story pass.
This may cause other Stories to break. At this point, you should not change those Stories (unless they are directly affected), but instead, fix the specs in the method outlined earlier, when looking at an internal change.

If your Stories are comprehensive enough, these processes will stop you from introducing bugs, while at the same time allowing you to spec the “right way” (at least according to me): by specifying behaviours rather than outcomes.

Issues with speccing validators and relationships

There are still some issues to do with validators and relationships. How should they be specced? The arguments tend to go like this:

Someone writes code like:

##################
describe User, "with neither facebook uid nor phone number set" do
  include UserSpecHelper

  before(:each) do
    @user = User.new
  end

  it "should not be valid" do
    @user.should_not be_valid
  end
end

##################
describe User, "with facebook uid" do
  include UserSpecHelper

  before(:each) do
    @user = User.new(valid_facebook_attributes)
  end

  it "should be valid" do
    @user.save!
    @user.should be_valid
  end

  it "should reject duplicate facebook uid" do
    @user.save
    @user_2 = User.new(valid_facebook_attributes)
    @user_2.should_not be_valid
  end
end

This prompts someone else to ask:

1 - What are you supposed to be testing?

The answer to this is usually “your own code”. Therefore, a test (I refuse to call it a specification) like the above makes no sense. Yet is the most common way that people specify a validator.

What am I testing here? The validator code. That’s really pointless, because Rails already tests that. And it results in a lot of duplication (though technically, it can be argued to be “not duplication” because you’re testing that the business rule of requiring a particular field is there, but I think that’s just rationalising a bad practice). Ultimately, though, this is uncomfortable because you’re testing an outcome, not specifying a behaviour. You may use rspec to do so, but it’s still an outcome. And that’s not what behaviour driven design is about. Ultimately, this is testing, not specification.

2 - But how do a specify one of those weird validator things?

I think a lot of the issue is also caused by the simple language constraint that specifying validators and relationships is just plain fugly at the moment. Here’s the code provided by David Chelimsky (chief maintainer of rspec):

it "should validate_presence_of digits" do
  PhoneNumber.expects(:validates_presence_of).with(:digits)
  load "#{RAILS_ROOT}/app/models/phone_number.rb"
end

Now I love rspec, and very much respect David - I’ve had a few chats with him on freenode#rspec and he’s a very smart guy - but this code is just nasty! That nastiness is required to get around the fact that the validator is loaded when the class loads, so can’t be specified as something that will happen later.

As per the Sapir-Whorf hypothesis that BDDers know and love, language drives behaviour, and nasty language like this doesn’t encourage people to go this way, even though, as I’ll explain a bit further down, I think this is actually the correct way to go, in spirit if not in form.

3 - But if I just specify that “thing”, how do I know that it actually does what it’s supposed to?

A couple of weeks ago I would have replied: This is not a valid concern. It just shows that you really need to write that spec. You need to find out how your code is going to work before you write it. Surely that should be obvious?

But then, like all things, this is not quite so black and white. There are validation and relationship constructs which are complex enough that you want to not only specify them, you also want to unit test that they work, and that they continue working even when things change, for instance when you deploy Rails 2.0. And that’s the key to solving this puzzle. The “specification” in point 1 looks wrong because it’s not a specification, it’s a unit test. It’s not an integration test, so it doesn’t fit in the Story Writer. But it’s also not a specification, so it doesn’t fit in your User spec.

So, the solution?

I put forward that this item belongs in a third category, that we thought eliminated by the arrival of BDD, but which actually should still exist: unit tests. Specifically, unit tests relating to the behaviour of third party code, or of your interpretation of that code. So these are not system integration tests, in the sense that they are not testing your system as a whole. They are more like unit integration tests, there to codify your understanding of the way the rest of the world behaves.

As such, they do not fit in the Story Runner. They have nothing to do with user requirements. What we need is a new directory in the rspec folder, dedicated to what I would like to call Assumption Tests. They are tests of your assumptions about your dependencies. As such, they are both redundant and essential. They are redundant in an ideal world where you know exactly how the external code you use works and you know about any changes to it. They are essential in the ugly world where you have to be just that little bit paranoid about other people’s code, even if it’s specified and tested up the wazoo, because sometimes things change in a way that breaks your assumptions.

And, importantly, these are not specifications. They are unashamedly outcome-driven. This is especially acceptable when you consider that they are not specifying your own code, but testing other people’s code to make sure it fits your assumptions.

Without further ado, here’s an example of one such assumption test:

class ActiveRecordTester < ActiveRecord::Base

  # This makes this model tableless
  def self.columns() @columns ||= []; end
  def self.column(name, sql_type = nil, default = nil, null = true)
    columns << ActiveRecord::ConnectionAdapters::Column.new(name.to_s, default, sql_type.to_s, null)
  end

  column :foo, :string

  validates_presence_of :foo
end

##################
describe "Presence Validator" do

  it "should not be valid without foo" do
    @test = ActiveRecordTester.new
    @test.should_not be_valid
  end

  it "should be valid with foo" do
    @test = ActiveRecordTester.new(:foo => "1")
    @test.should be_valid
  end

end

Once you’ve got this in place, you should never need to test the outcome of validatespresenceof again. So the code in example 1 is now obsolete, and your specifications can remain entirely behaviour-driven, rather than including bits and pieces of outcome testing. All you need to specify is that, say, your User model calls “validatespresenceof” for facebook_uid in the way that’s codified and auto-tested in your assumptions.

So now you have:

specs for controllers
specs for models
specs for views
stories for the whole system
assumption tests for things which you didn’t write but want to test anyway

Sorry for the long article, but there were a fair number of ideas to cover. I’d love to hear comments about this, as it’s all still a bit fresh to me. You can put those comments below, or even comment on your own blog (I have trackback enabled). Thanks for reading!

Specifying a before_filter and/or private method

Thu, 04 Oct 2007 13:12:00 +0000

How do you write specs for a private or protected method? Specifically (as this is often the case), how do you write a spec for a before_filter?

The answers that I found while looking online were not very explicit. The main points seemed to be “don’t test private/protected method”, “you should be specifying expected behaviours in the places where they’re used.” I decided to write this article to clarify how I’ve interpreted that advice.

Essentially, I decided to follow the advice given online. Don’t test the methods, test the behaviours you expect. Let’s take a simple example and see what this means in practice.

For our facebook application, I have a facebook controller which handles all the facebook posts (very RESTless, but that’s facebook for you!). There are two things that I want to see happen with every authenticated facebook action, and I’ve defined a filter for each of them:

class FacebookController < ApplicationController
  before_filter :require_facebook_install
  before_filter :set_user

  def index
    # ...
  end

  def other_action
    # ...
  end

private
  def set_user
    if @user = User.find_by_facebook_uid(fbparams["user"], :include => [:interactions])
      if fbparams["friends"]
        @user.cached_friend_uids = fbparams["friends"]
        @user.save
      end
      if fbparams["session_key"]
        @user.facebook_session_key = fbparams["session_key"]
        @user.save
      end
    end
  end

end

Now, requirefacebookinstall is given by RFacebook, so all I want to specify is that for certain actions, this filter must be called. All that it does is return true if proper parameters are included, that represent a valid facebook session. There’s a lot of those parameters, and I only use a few of them. The only time when I might want to have this filter return false is if I was testing what would happen with a blank session. However, this should never happen in my specifications, since I’m not trying to spec RFacebook, but my own code. This is the simple, shared spec that I wrote:

describe "Facebook Action", :shared => true do
  include FacebookControllerSpecHelper

  before(:each) do
    @controller.should_receive(:require_facebook_install).and_return(true)
  end

end

It’s not much to look at, and it doesn’t need to be! How to use it? Just include this (which I’d like to call a partial spec, but the rspec keyword is “shared”) into other specs:

describe FacebookController, "index page" do
  include FacebookControllerSpecHelper

  it_should_behave_like "Facebook Action"

  before(:each) do
    @user = User.create(:facebook_uid => 1)
  end

  def do_post
    post :index
  end

  it "should render the canvas" do
    do_post
  end

end

I can do this for any of the other facebook “pages”, of course. So far so good. Adding this “itshouldbehavelike” directive ensures that the filter is called for the actions where I specify that it needs to be called. Handily enough, this also removes the need for me to mock requirefacebook_install in every one of those specs, since it’s mocked just once in the shared spec.

How about setuser? That’s a little bit more tricky. In this case, I have actually written setuser myself, so I need to specify it. But I can’t! It’s a private method, therefore invisible to my specs! Oh dear. Let’s go back to the initial advice. Don’t specify methods, specify behaviours where they’re used. Okay. Thinking about it on a basic level, this makes sense. I do want to specify that my filter is working for each of the actions I’ve told it to work for. One way to do this would be to specify that set_user must function properly in each of those actions. The smart way is of course to put that into a partial spec as well.

describe "Facebook Action With User", :shared => true do
  include FacebookControllerSpecHelper

  before(:each) do
    @user = User.create(:facebook_uid => valid_fbparams["user"])

    @controller.should_receive(:fbparams).at_least(:once).and_return(valid_fbparams)
    User.should_receive(:find_by_facebook_uid).at_least(:once).and_return(@user)
  end

  it "should call set_user" do
    do_post
  end

  it "should save the user's cached friends" do
    @user.should_receive(:cached_friend_uids=).with(valid_fbparams["friends"])
    @user.should_receive(:save).at_least(:once)

    do_post
  end

  it "should save the user's session key" do
    @user.should_receive(:facebook_session_key=).with(valid_fbparams["session_key"])
    @user.should_receive(:save).at_least(:once)

    do_post
  end
end

Those are the three things I expect my setuser beforefilter to do for me. So now I can include them in my controller action spec as well!

describe FacebookController, "index page" do
  include FacebookControllerSpecHelper

  it_should_behave_like "Facebook Action"
  it_should_behave_like "Facebook Action With User"

  def do_post
    post :index
  end

  it "should render the canvas" do
    @controller.instance_variable_set(:@user, @user)
    do_post
  end

end

And that’s all. The best part is, including the “Facebook Action With User” behaviour automatically mocks that part of the process for me in all the actions where I put it, so that I don’t need to do it again and again. By extracting those apparently intractable filters into their own shared spec, I’ve not only managed to spec them, but I’ve also reduced the clutter in my mocking code. I now only mock the things which are relevant to the current action, instead of having to mock everything to “make things work”.

On the downside, by DRYing up this code it’s become a tad harder to follow at a glance, but I think it’s more readable once you’ve figured out what the included shared specs provide.

One final caveat: if you mock something in your shared specs, don’t mock it again in your concrete specs. Otherwise, you will get errors about your shared spec being broken.

I hope you found this article useful. Please feel free to leave a comment below if you agree or disagree with this method - particularly if you have something better to suggest!

The difference between TDD and BDD

Wed, 03 Oct 2007 08:59:00 +0000

“It’s all about the words”. “Focus on behaviour”. “Design your API as a client of your own API”. “It’s a clearer way to express the same thing, so it helps communication with stakeholders”. Those are all typical responses to the questions “Why is Behaviour Driven Development better than Test Driven Development? Why should I switch to BDD?”

I’ve given this some thought. All these reasons are correct, of course. BDD uses different wording, which is much clearer. BDD focuses on specifying behaviour (though the benefit of that is often not clarified, and some people will claim that all you’re doing is writing your code twice). Designing your own API as your own client means you will design it more easily and more quickly. Being clearer, BDD does help to communicate with other coders or stakeholders (although I’m not entirely sure that the average spec is really client-friendly enough to show unfiltered).

Some of those are not really benefits of BDD, however - rather, they are benefits of Designing. Many programmers (particularly in the less sophisticated parts of the agile movement) tend to focus on code and ignore the rest. BDD provides them with a neat, coder-friendly way of doing design and specification. Of course that helps! There’s a reason why the industry standard development processes include a design phase, after all - it’s not just to delay the project!

I think there’s another important reason why BDD is better, as a test suite. It helps make your test suite less brittle in the face of specification changes. Here’s the reasoning.

Update: Please have a look at this article from Pat Maddox. BDD is a fairly new field that’s still evolving, and Pat’s article is right on the money. The approach I outline in this article is what Pat would call “pure interaction testing” and, as he points out, it should not be a goal - only an available technique. Currently, I use pure interaction testing for Rails controller testing, and state-based testing for ActiveRecord models. There’s certainly a place for the ideas exposed in this article, but they shouldn’t be used blindly everywhere.

BDD focuses on specifying what will happen next. The basic sentence structure of a BDD spec is:

This should happen; then this should happen; then this. Tell me if that doesn’t happen when I say “foo”. Foo.

TDD, on the other hand, focuses on setting up a set of conditions and then looking at the output:

Foo. Is Bar set to the right value? What about Baz?

Put in other words, BDD specifies behaviours while TDD specifies outcomes. The difference is more than just language, though. It’s a paradigm shift that has some severe consequences.

The main one, in my view, is that behaviours can be layered, while outcomes cannot. This might sound a bit cryptic, so let me explain. Let’s use the analogy of a big machine. At one end, you put a variety of raw materials, and at the other, you get a widget. Outcome testing will examine the widget from all angles. It will ask “Does this widget perform the functions that I expect a widget to perform?”. This is basically black box testing - and that’s what most integration tests do (and, sadly, many unit tests, as it’s often hard to separate the two without extensive mocking, which requires deep knowledge of the internals of the module you’re testing).

Behaviour specification turns this upside down. Instead of trying to specify the outcome, and using mocking only where you absolutely need to in order to get the thing working, you start with the goal to specify precisely those internals that you tried to avoid in TDD. “The machine should be able to turn sand and gold flakes into gold flaked glass; to do this, it should heat up the sand into molten glass. Then it should insert the gold flakes.” This requires internal knowledge of the machine.

Hopefully this is clear so far. Now here’s the clincher, the real advantage of behaviour specification over outcome testing. In order to test an outcome, you need all the parts up to that outcome to have done their job. If I want to test that my widget has a gold-flaked glass pane in it, I need the rest of the widget-building to have succeeded and not produced a molten lump of goo. In other words, outcome testing is quite brittle. If one of the parts malfunctions, there will be many knock-on effects onto other tests. You might point out that this is why we have unit tests, but my point is, BDD enables you to design your unit tests properly. Let’s take a fairly simple method as an example.

class Mogrifier < Polator
  def transmogrify(foo, bar)
    foo.extrapolate!
    bar.extrapolate!

    baz = intrapolate(foo, bar)

    return ultrapolate(baz)
  end
end

With a Test-First approach, we might have written something like this as a test, before writing this code:

def test_mogrifier
  foo = test_foo
  bar = test_bar

  test_result = Mogrifier.new.transmogrify(foo, bar)

  assert test_result.ultrapolated?
  assert test_result.intrapolated?
  assert test_result.contains(foo.relevant_bit)
  assert test_result.contains(bar.relevant_bit)
end

How would you write this as a specification?

describe Mogrifier, "transmogrify" do

  before(:each) do
    @mogrifier = Mogrifier.new
  end

  it "should extrapolate foo" do
    foo = "foo"
    foo.should_receive(:extrapolate!).and_return("extrapolated foo")

    @mogrifier.my_method(foo, "bar")
  end

  it "should extrapolate bar" do
    bar = "bar"
    bar.should_receive(:extrapolate!).and_return("extrapolated bar")

    @mogrifier.my_method("foo", bar)
  end

  it "should intrapolate" do
    @mogrifier.should_receive(:intrapolate)
    @mogrifier.my_method("foo", "bar")
  end

  it "should ultrapolate" do
    @mogrifier.should_receive(:ultrapolate)
  end

end

Why is the test more brittle? It’s brittle because if the specification changes and states that actuallly, the transmogrifier should also encrypt its results, your test will break, whereas not one of your specs will break. In a BDD world, you’d write:

  it "should encrypt" do
    mock_baz = mock("baz")
    mock_baz.should_receive(:encrypt)
    @mogrifier.should_receive(:ultrapolate).and_return(mock_baz)
  end

And then you’d add the encrypt call to the transmogrify method. With TDD, on the other hand, any of your tests that depend on the outcome of the transmogrifier will have to change. You might even be quite stuck at this point, in this specific example, if the encryption happens to be one-way - if that were the case, you won’t be able to test the outcome at all, since it will be opaque! Then you’d have to redesign your code so that transmogrify doesn’t actually encrypt stuff, but some other method does. What a pain!

Behaviour specifications enable you to ensure that your transmogrifier code keeps extrapolating and intrapolating as required - no matter what it does with its input later. If it needs to do other stuff too, that’s fine, you can specify it as well - but each specification will (or should) focus on a single behaviour and that way, as long as your spec passes, you know that the specification is being met, and that foo and bar are still being extrapolated (in this example).

As the behaviours multiply, and the layering increases, this becomes even more useful. Because of the focus of each specification, there is rarely any need to change more than a handful of specifications when the code changes (if the specs are well-written and cleanly split out into small and specific behaviours). For instance, if the Polator changes its processes (but retains its interface), the Transmogrifier test code will remain the same. BDD encourages low coupling of tests, which, to be fair, TDD doesn’t encourage. In TDD, it’s easier to write a test that uses real objects (at least at first). In BDD, you simply cannot specify behaviours without using mocks.

This is the major winning point of BDD for me, the reason why it feels so much more right. I think that the clarity, the speed, design, words, etc, are all nice bonuses to this main benefit. If you have any comments on this, any disagreements, etc, do feel free to post something below!

PS:

On the other hand, please note that I don’t suggest that we shouldn’t use tests at all. As mentioned here, specs do not validate whether your system actually hangs together and works on the whole. Tests are the ones to give you that.