ruby unit test frameworks « the evolving ultrasaurus

In preparation for teaching Ruby in a class with test first teaching. I decided to evaluate a few test frameworks. I thought initially to use Test::Unit, since it seemed easy to understand and ships with Rails. Wolfram Arnold argued that Test::Unit would burden the new folks with legacy. Alex Chaffee also advocated RSpec, but other friends from the Twittervese had good things to say about shoulda. Some folks declared it to be simply a matter of taste.

Even so, I wanted to make an informed decision and refine my palette for Ruby tools, so I wrote a simple exercise in each of Test::Unit, Shoulda and RSpec.

Test::Unit

require 'test/unit'
require 'pig_latin'
 
class PigLatinTest < Test::Unit::TestCase
    include PigLatinTranslator
 
    def test_simple_word
        s = translate("nix")
        assert_equal("ixnay", s)
    end
 
    def test_word_beginning_with_vowel
        s = translate("apple")
        assert_equal("appleay", s)
    end
 
    def test_two_consonant_word
        s = translate("stupid")
        assert_equal("upidstay", s)
    end
end

With the above code saved as “test_pig_latin.rb” you run it by simply executing it with Ruby.

$ ruby test_pig_latin.rb
Loaded suite test_pig_latin
Started
FFF
Finished in 0.01091 seconds.

  1) Failure:
test_simple_word(PigLatinTest) [test_pig_latin.rb:9]:
 expected but was
.

  2) Failure:
test_two_consonant_word(PigLatinTest) [test_pig_latin.rb:19]:
 expected but was
.

  3) Failure:
test_word_beginning_with_vowel(PigLatinTest) [test_pig_latin.rb:14]:
 expected but was
.

3 tests, 3 assertions, 3 failures, 0 errors

Shoulda

Notice in the code below that Shoulda is simply and extension to Test::Unit. The PigLatinTest also subclasses Test::Unit::TestCase, just as the example above; however, the code inside the test case looks substantially different (and more readable in my opinion). You can actually mix Shoulda tests (below) with regular TestCase test methods (above) in the same TestCase. This is an advantage to Shoulda over RSpec if you have a codebase that already has lots of unit tests; however, I have also used RSpec and Test::Unit in the same project (you just have to remember to ‘rake test’ and ‘rake spec’).

require 'rubygems'
require 'shoulda'
require 'pig_latin'

class PigLatinTest < Test::Unit::TestCase
  include PigLatinTranslator

  context "#translate" do

    should "translate a simple word: nix" do
      s = translate("nix")
      assert_equal("ixnay", s)
    end

    should "translate a word beginning with a vowel: apple" do
      s = translate("apple")
      assert_equal("appleay", s)
    end

    should "translate a two consonent word: stupid" do
      s = translate("stupid")
      assert_equal("upidstay", s)
    end

  end
end

With the code above saved as “test_shoulda_pig_latin.rb” you use the same process as above by just executing the file with ruby.

$ ruby test_shoulda_pig_latin.rb
Loaded suite test_shoulda_pig_latin
Started
FFF
Finished in 0.008268 seconds.

 1) Failure:
test: #translate should translate a simple word. (PigLatinTest)
 [test_shoulda_pig_latin.rb:12:in `__bind_1251676444_52936'
 /Library/Ruby/Gems/1.8/gems/thoughtbot-shoulda-2.10.2/lib/shoulda/context.rb:351:in `call'
 /Library/Ruby/Gems/1.8/gems/thoughtbot-shoulda-2.10.2/lib/shoulda/context.rb:351:in `test: #translate should translate a simple word. ']:
<"ixnay"> expected but was
<"translation">.

 2) Failure:
test: #translate should translate a two consonent word. (PigLatinTest)
 [test_shoulda_pig_latin.rb:22:in `__bind_1251676444_58860'
 /Library/Ruby/Gems/1.8/gems/thoughtbot-shoulda-2.10.2/lib/shoulda/context.rb:351:in `call'
 /Library/Ruby/Gems/1.8/gems/thoughtbot-shoulda-2.10.2/lib/shoulda/context.rb:351:in `test: #translate should translate a two consonent word. ']:
<"upidstay"> expected but was
<"translation">.

 3) Failure:
test: #translate should translate a word beginning with a vowel. (PigLatinTest)
 [test_shoulda_pig_latin.rb:17:in `__bind_1251676444_59935'
 /Library/Ruby/Gems/1.8/gems/thoughtbot-shoulda-2.10.2/lib/shoulda/context.rb:351:in `call'
 /Library/Ruby/Gems/1.8/gems/thoughtbot-shoulda-2.10.2/lib/shoulda/context.rb:351:in `test: #translate should translate a word beginning with a vowel. ']:
<"appleay"> expected but was
<"translation">.

3 tests, 3 assertions, 3 failures, 0 errors

RSpec

require "pig_latin"

describe "#translate" do
  include PigLatinTranslator

  it "should translate a simple word" do
    s = translate("nix")
    s.should == "ixnay"
  end

  it "should translate a word beginning with a vowel" do
    pending
    s = translate("apple")
    s.should == "appleay"
  end

  it "should translate a two consonent word: stupid" do
    pending
    s = translate("stupid")
    s.should == "upidstay"
  end

end

The code above is saved in a file called “pig_latin_spec.rb” and run it using the ‘spec’ command. You will need to have installed the rspec gem (sudo gem install rspec).

$ spec pig_latin_spec.rb
F**

Pending:

#translate should translate a word beginning with a vowel (TODO)
./pig_latin_spec.rb:11

#translate should translate a two consonent word: stupid (TODO)
./pig_latin_spec.rb:17

1)
'#translate should translate a simple word' FAILED
expected: "ixnay",
     got: "translation" (using ==)
./pig_latin_spec.rb:8:

Finished in 0.035728 seconds

3 examples, 1 failure, 2 pending

Conclusion

I like RSpec best since I find the output to be most readable. I love the pending keyword, which allows me to set up the tests as an exercise for the class with only one test failing. I find it helps focus on exactly one test and one failure. I considered going with Shoulda because the tests are just as readable as RSpec, even if the output takes some learning to read, because of my initial thought that Test::Unit held less magic. However, on closer inspection, I realized that Test::Unit has one significant magical incantation: you merely declare a class and when that class is defined, it runs the test. This seemed not the kind of topic I would want to teach in an intro class. Even some experienced programmers might struggle with understanding the mechanism that allows such a construct to function. I concluded that all of the test frameworks require serious magic, and picked RSpec since I found it to be most usable for test writing and analysis of the output.

Caveat: this exercise was for pure Ruby. In Rails, I wonder if Shoulda tests would be more concise, making them easier to write and read and, therefore, making it worth the steeper learning curve on reading the output.

12 thoughts on “ruby unit test frameworks”

I always find myself sticking to Test::Unit just because it’s included with Ruby by default. I think once I’ve mastered the ins and outs of Test::Unit, then I’ll move onto something like Shoulda or RSpec.

I guess my pig-latin is rusty. I thought that would be
it “should translate a word beginning with a vowel” do
pending
s = translate(“apple”)
s.should == “applevay”
end

Wow. That’s a different dialect than I learned. Wikipedia notes that there are different consonants used in different dialects:

“In words that begin with vowel sounds or silent consonants, the syllable “ay” is added to the end of the word. In some dialects, to aid in pronunciation, an extra consonant is added to the beginning of the suffix; for instance, eagle could yield eagle’yay, eagle’way, or eagle’hay.”

http://en.wikipedia.org/wiki/Pig_Latin#Rules_and_variations

If anyone wants to experiment with the files at home, I posted them on github: http://github.com/ultrasaurus/test-framework-comparison/tree/master

You can actually use Test::Unit in RSpec (http://blog.davidchelimsky.net/2009/2/2/rspec-works-with-test-unit/) and Shoulda has some special RSpec hooks as well. I’ve found this useful when incrementally transforming Test::Unit tests into RSpec examples.

You’ll find the same Rspec look/feel of code and output from within Rails.
There is one spot i notice that still falls apart, no matter the test framework : diffs of failing array/hash comparison. Anyone have a Rpec ‘plugin’ for this?

Peter, I’m doing this in my current app to compare many deep structures: http://gist.github.com/287675

Not elegant but it works.

I’ve been using Shoulda in Rails for a couple of years, and I don’t really use its Rails-specific test methods (e.g., #should_belong_to). They’ve always seemed just a little too tautological for my taste — in that they don’t describe actual value. If there aren’t other tests that fail without a given association being declared, You’re Doing It Wrong ™.

That being said, I do like Shoulda’s nested contexts, if for no other reason than that they let me fold up large sections of tests that I don’t currently care about.

If it’s a matter a taste, and one of the options comes with Ruby, then that’s the obvious choice. Especially when teaching!

Since when is the canon just legacy?

If people want to switch to some other flavor they like, great! But they should be doing that based on real reasons they understand. It’s a major disservice to send them off with some Fav Flav. Even if it’s your fav flav.

I think we should teach using what is best for the student. As teachers, we need to use our best judgement in picking what collection of tools to introduce students to. This is not just a chocolate or strawberry kind of choice. I tried to explain in my post why I chose RSpec for teaching. As I mentioned, I actually started the experiment with a bias toward Test::Unit.

Right now minitest comes with Ruby and Test::Unit comes with Rails. Those projects pick their test frameworks for a lot of reasons — my guess is that being able to create very clear failing tests was not a high priority. In the case of Rails, RSpec didn’t exists when they chose Test::Unit and they are working on improving the readability of output (https://github.com/TwP/turn) but I still think it doesn’t approach the clarity of RSpec.

Pingback: on choosing RSpec as a test framework | the evolving ultrasaurus

YMMV, of course, but I use test::Unit over rspec because altering the output messaging is actually far easier.

Sure, for very simple tests like your examples – that use the default messaging, the output seems better. But if you want to pass custom failure messages, rspec requires you to declare a brand new class every time… whereas Test::Unit you can just pass it as a param.

This becomes more important in actual, complex tests. I have a lot of asserts that are sanity checks… and I like to make sure my code outputs *what* failed and what it should have been.. with a comment eg as a toy example:

should “approve a new widget” do
w = MyWidget.new(:name => “my widget”, :approved => false)
assert !w.approved?, “sanity check: test data should not have started out approved”
assert w.approve!, “approval process should have worked”
assert w.approved? # note – no need to comment this one as it’s the *actual* test
end

The last line is the actual thing you’re testing – but sanity checking the data and the intermediate steps is useful to make sure you are *actually testing what you think you’re testing* – something that you often only find out later when it breaks…
and having better output on the lines makes Test::unit far easier to get proper messages passed around as to what should be happening and why.

Matthew Lang said on August 31, 2009 at 1:23 pm:

I always find myself sticking to Test::Unit just because it’s included with Ruby by default. I think once I’ve mastered the ins and outs of Test::Unit, then I’ll move onto something like Shoulda or RSpec.
Adam Wolff said on August 31, 2009 at 4:28 pm:

I guess my pig-latin is rusty. I thought that would be
it “should translate a word beginning with a vowel” do
pending
s = translate(“apple”)
s.should == “applevay”
end
Sarah said on August 31, 2009 at 7:48 pm:

Wow. That’s a different dialect than I learned. Wikipedia notes that there are different consonants used in different dialects:

“In words that begin with vowel sounds or silent consonants, the syllable “ay” is added to the end of the word. In some dialects, to aid in pronunciation, an extra consonant is added to the beginning of the suffix; for instance, eagle could yield eagle’yay, eagle’way, or eagle’hay.”

http://en.wikipedia.org/wiki/Pig_Latin#Rules_and_variations
Sarah said on August 31, 2009 at 7:55 pm:

If anyone wants to experiment with the files at home, I posted them on github: http://github.com/ultrasaurus/test-framework-comparison/tree/master
Jeremy D. Frens said on September 1, 2009 at 5:08 am:

You can actually use Test::Unit in RSpec (http://blog.davidchelimsky.net/2009/2/2/rspec-works-with-test-unit/) and Shoulda has some special RSpec hooks as well. I’ve found this useful when incrementally transforming Test::Unit tests into RSpec examples.
Peter Fitzgibbons said on September 1, 2009 at 1:32 pm:

You’ll find the same Rspec look/feel of code and output from within Rails.
There is one spot i notice that still falls apart, no matter the test framework : diffs of failing array/hash comparison. Anyone have a Rpec ‘plugin’ for this?
Scott Bronson said on January 27, 2010 at 1:01 am:

Peter, I’m doing this in my current app to compare many deep structures: http://gist.github.com/287675

Not elegant but it works.
Sam Livingston-Gray said on March 31, 2011 at 9:14 am:

I’ve been using Shoulda in Rails for a couple of years, and I don’t really use its Rails-specific test methods (e.g., #should_belong_to). They’ve always seemed just a little too tautological for my taste — in that they don’t describe actual value. If there aren’t other tests that fail without a given association being declared, You’re Doing It Wrong ™.

That being said, I do like Shoulda’s nested contexts, if for no other reason than that they let me fold up large sections of tests that I don’t currently care about.
Paris Sinclair said on March 31, 2011 at 10:47 am:

If it’s a matter a taste, and one of the options comes with Ruby, then that’s the obvious choice. Especially when teaching!

Since when is the canon just legacy?

If people want to switch to some other flavor they like, great! But they should be doing that based on real reasons they understand. It’s a major disservice to send them off with some Fav Flav. Even if it’s your fav flav.
Sarah said on March 31, 2011 at 2:48 pm:

I think we should teach using what is best for the student. As teachers, we need to use our best judgement in picking what collection of tools to introduce students to. This is not just a chocolate or strawberry kind of choice. I tried to explain in my post why I chose RSpec for teaching. As I mentioned, I actually started the experiment with a bias toward Test::Unit.

Right now minitest comes with Ruby and Test::Unit comes with Rails. Those projects pick their test frameworks for a lot of reasons — my guess is that being able to create very clear failing tests was not a high priority. In the case of Rails, RSpec didn’t exists when they chose Test::Unit and they are working on improving the readability of output (https://github.com/TwP/turn) but I still think it doesn’t approach the clarity of RSpec.
Pingback: on choosing RSpec as a test framework | the evolving ultrasaurus
Taryn East said on July 23, 2011 at 7:03 am:

YMMV, of course, but I use test::Unit over rspec because altering the output messaging is actually far easier.

Sure, for very simple tests like your examples – that use the default messaging, the output seems better. But if you want to pass custom failure messages, rspec requires you to declare a brand new class every time… whereas Test::Unit you can just pass it as a param.

This becomes more important in actual, complex tests. I have a lot of asserts that are sanity checks… and I like to make sure my code outputs *what* failed and what it should have been.. with a comment eg as a toy example:

should “approve a new widget” do
w = MyWidget.new(:name => “my widget”, :approved => false)
assert !w.approved?, “sanity check: test data should not have started out approved”
assert w.approve!, “approval process should have worked”
assert w.approved? # note – no need to comment this one as it’s the *actual* test
end

The last line is the actual thing you’re testing – but sanity checking the data and the intermediate steps is useful to make sure you are *actually testing what you think you’re testing* – something that you often only find out later when it breaks…
and having better output on the lines makes Test::unit far easier to get proper messages passed around as to what should be happening and why.

the evolving ultrasaurus

Sarah Allen's reflections on internet software and other topics

ruby unit test frameworks

Test::Unit

Shoulda

RSpec

Conclusion

12 thoughts on “ruby unit test frameworks”

Leave a reply Cancel reply