Unit Testing in C++ and Objective-C just got easier | Hacker News Reader

Unit Testing in C++ and Objective-C just got easier

levelofindirection.com

48 pointsbarredo16 comments

Google's gtest (and gmock) is a wonderful piece of C++ code. It needs very small amount boilerplate, less than most junit based frameworks (which is amazing, given this C++ that we are talking about) and great mocking abilities (for C++). I don't see how this CATCH can match it anytime soon. Headers-only approach is interesting but can be bad for compilation performance, which is very very important when projects grow in size.

The headers are included only in test code which is usually in separate compilation units, so the compile times shouldn't be a problem

Arguably compilation time of your test code is even more important! You don't want to be kept waiting for feedback in a TDD cycle. I agree that compilation time can be a limiting factor of the header-only approach (actually link time may be the killer). But it's generally easier to give a header-only library support for libs than the other way around. I plan to keep header-only as the default way of using CATCH.

I hear you on the compilation time. I've not yet found it to be a problem but I haven't used it at large scales yet. I'm considering support for a lib based alternative version for larger scale use (I believe boost.test has the same approach).

As for gtest, I believe I already a decent amount of the same ground. In same cases I do things different ways, so it's difficult to compare directly. In others it may be simply a matter of "coming soon".

I don't see anything in gtest that is not already supported or readily achievable in CATCH.

What I'm not planning on supporting directly is mocking support - but there is no reason why gmock couldn't be used here.

If your project is template intensive you're going to have long compilation time.

Our work around is parallel compilation.

What impresses me is this:

The assertion macro is REQUIRE( expression ) [...] Don't worry - lhs and rhs are captured anyway - more on this later.

Phil wrote a bit about this technique -- capturing the two sides of an assertion -- here: http://www.levelofindirection.com/journal/2010/5/21/the-ulti...

Very interesting trick. Basically when you write something like

  CHECK(a == b);

the expression-capturing part of the code is something like (these are not the actual names)

  #define CAPTURE(expr) SomeStrangeObject() ->* expr

which for a == b expands to something like

  SomeStrangeObject() ->* a == b

Since ->* has highest priority and is seldom overloaded, this is actually parsed by the compiler as

  (SomeStrangeObject() ->* a) == b

The subexpression in parentheses returns an object which wraps a and overloads the comparison operators (==, >, < ...), so that it can capture also the comparison operator used and b.

While very convenient it is kind of easy to "break". For example

    int one = 2;
    CHECK( one == 1 );
    CHECK( (one == 1) );

the two checks are seemingly the same, but in the second case the trick doesn't work:

  my_test.cpp(6): one == 1 failed for: 2 == 1
  my_test.cpp(7): (one == 1) failed for: 0

Also, expressions not in the '<operand> <comparison operator> <operand>' form are not supported, for example

    CHECK ( one + 1 == 2 );

causes this obscure error message

  my_test.cpp:10: error: no match for ‘operator+’ in ‘Catch::ResultBuilder(((const char*)"one + 1 == 2"), 0, ((const std::string&)(& std::basic_string<char, std::char_traits<char>, std::allocator<char> >(((const char*)"my_test.cpp"), ((const std::allocator<char>&)((const std::allocator<char>*)(& std::allocator<char>())))))), 10u, ((const std::string&)(& std::basic_string<char, std::char_traits<char>, std::allocator<char> >(((const char*)"CHECK"), ((const std::allocator<char>&)((const std::allocator<char>*)(& std::allocator<char>()))))))).Catch::ResultBuilder::operator->* [with T = int](((const int&)((const int*)(& one)))) + 1’

Same for

    CHECK( 1 == 1 && one == 1);

which fails with

  my_test.cpp:9: error: no match for ‘operator&&’ in ‘((Catch::ResultBuilder*)Catch::ResultBuilder(((const char*)"1 == 1 && one == 1"), 0, ((const std::string&)(& std::basic_string<char, std::char_traits<char>, std::allocator<char> >(((const char*)"my_test.cpp"), ((const std::allocator<char>&)((const std::allocator<char>*)(& std::allocator<char>())))))), 9u, ((const std::string&)(& std::basic_string<char, std::char_traits<char>, std::allocator<char> >(((const char*)"CHECK"), ((const std::allocator<char>&)((const std::allocator<char>*)(& std::allocator<char>()))))))).Catch::ResultBuilder::operator->* [with T = int](((const int&)((const int*)(&1)))))->Catch::ResultBuilder::operator== [with RhsT = int](((const int&)((const int*)(&1)))) && (one == 1)’
  my_test.cpp:9: note: candidates are: operator&&(bool, bool) <built-in>

Not sure if these issues can cause real problems in everyday use, but they may cause headaches if one doesn't want to dig into preprocessor/template metaprogramming code.

You make a good point and I have put in some workarounds for this already. I'll probably put more in over time. The trade-off is that I didn't want to over-use expression templates so I have kept it to one level. But my aim is to at least allow arbitrary expressions to compile and correctly report failures. It should also detect when the expression cannot be decomposed and report this in the output so you can rewrite the expression if you do hit it. I have done this for some operators already.

I want to say mine are nitpickings anyway, the library is very nice and I share your dislike for complex unit test libraries. I had to use the Boost Testing framework recently and that was a pain.

Don't know if it is relevant, but I think you can also work around the parentheses problem using some preprocessor magic. Boost Preprocessor does something similar in its handling of lists, but I haven't been using it for a very long time and I don't remember how it did the trick.

I really like INFO(); that's a nice feature I've not seen anywhere else.

What are the differences between a C++ and Objective-C test framework? Is there anything ObjC-specific about CATCH?

Some of the other design choices here I've used, and later abandoned, in my own testing libraries:

> The assertion macro is REQUIRE( expression ), rather than the, now traditional, REQUIRE_EQUALS( lhs, rhs ), or similar. Don't worry - lhs and rhs are captured anyway - more on this later.

Using C++ operators for assertions is very limiting; some interesting assertions, like floating-point near-equality, have no operator and thus must be tested separately. I like to use separate functions to build "assertion results", so you can get code equivalent to:

  REQUIRE(equal(lhs, rhs));
  REQUIRE(equal_within(lhs, rhs, 0.001));
  REQUIRE(not_null(lhs));

and so on. With this style, you also get multi-variable assertions and value capture for free.

> We didn't name the function. We named the test case. This frees us from couching our names in legal C++ identifiers. We also provide a longer form description that serves as an active comment

It can be nice to have a description of the test case, but free-form names like you have are a major pain to write out on the command line if you're running a particular set of tests during development. If you want a description, allow it in addition to the test name.

Floating point comparisons are achieved through the Approx helper class. E.g:

    REQUIRE( pi == Approx( 3.14159265358 ) );

(and yes, I know pi to a lot more digits than that!)

The style you suggest is almost supported out of the box, and I've been considering fully supporting it - especially for container oriented tests.

As to the test naming - the idea is that you provide a short, hierarchical name, and a longer description (so the description is in addition to the test name):

TEST_CASE( "example/test1", "Such and such should do so and so" )

To run a test you need only specify the name (example/test1) - or a part of the name (e.g. example/*) (allowing tests to be easily grouped into suites with no extra work).

Also interesting to read, a previous blog entry from May 21, where he explains the birth of this test framework: http://www.levelofindirection.com/journal/2010/5/21/the-ulti...

> Most require you to build libraries. This can be a turn off to anyone who wants to get up and running quickly - especially if you just want to try something out. This is especially true of exploratory TDD coding.

This is absolutely true, and especially painful/noticeable if you target multiple platforms. To work around this problem, I've started using the stand-alone version of the Boost Unit Test framework:

    #include <boost/test/included/unit_test.hpp>

The problem with including that, like many other Boost header-only libraries, is that they wreck havoc on compile time :/.

Reminds me of py.test http://pytest.org/assert.html#assert-with-the-assert-stateme..., which is a rather good thing :)

Interesting, but I want a demo of how mocks and fakes are handled.

Mocks and fakes are not directly supported, however if you are rolling your own CATCH allows test code to be called back by your application code. Although I haven't tried it yet there should be no problem using CATCH with a third-party mocking framework.

Reply on news.ycombinator.com