Hacker News Reader

Comment by rgoulter | Hacker News Reader

I can appreciate that an automated test that runs to completion will cover more code that a unit test which aborts early. And that using 'soft' asserts lets the test run further than a hard assert.

My question is more basic than that, though.

Perhaps to clarify my understanding: code coverage indicates what code was 'covered' by executed tests. This can be useful to show: what parts work as expected in at-least one case, and what parts of code haven't been covered by tests at all. -- But with code coverage from failing tests, you can't get either of "covered code will work as expected in at-least one case" or "this code isn't covered by tests" (since the code that fails the tests is shown as 'covered'). -- What value do you get from code coverage from failed tests?

View on HN

Short answer: Not all failed test coverage is invalid.

Granted, unit tests benefit less from soft assertions, but I'll take a shot at an example anyway:

Consider a complex object returned from a unit test result (this would be a single target state). That structure could have a missing field that should be reported, but the rest of the fields are evaluated fine, and so that information would be lost if the first assert halted the test.

For functional tests, the value is usually easier to discern, so for another contrived example:

A login form has a failed assert on it's UI structure, say a button is the wrong color, but the login is successful, and the following assertion post-login is also successful.

Ideally, you would assert post-login tests in separate target state, perhaps mocking the login as a pre-condition to another test. For functional tests that could dramatically increase the testing time and complexity, and so this makes a non-halting assertion valuable.

Ah, I misread "test coverage" as "code coverage". Whoops.

> Consider a complex object returned from a unit test result (this would be a single target state). That structure could have a missing field that should be reported, but the rest of the fields are evaluated fine, and so that information would be lost if the first assert halted the test.

Right, I'd understand this as 'soft' assertions provide an easier way of making a composite assertion. e.g. I think with RSpec, its matches can be composed; its matchers can nicely return whether a field is missing, or a custom matcher can be written. -- Either way, the point is that a test failure results in the signal that's wholly useful.

> [...say, for cases where an early part of the test fails in an unimportant way...]

Hmm. I think I'd take this suggestion as: test for 'insignificant' things as low as possible; it's expensive to halt a slow functional test over something that doesn't matter. -- Once something that matters fails, that's the information which is useful for the test to signal.

And maybe it can be useful to warn about insignificant things that do get picked up during the functional test.

I appreciate the conversation! That's interesting about RSpec, I'll read more about it, thanks.

Reply on news.ycombinator.com