Hacker News Reader

Comment by tracnar | Hacker News Reader

We have a test environment with this taken to the extreme at work, i.e. integration tests where the whole codebase softly fails and continues whatever happens. IMO the drawbacks are worse than the upside of seeing more errors. Now one errors ends up giving you a lot of failing soft assertions after the first failure which are not useful at all. There is certainly benefit to being able to go through a failure, but it seems better handled at the framework level rather than writing the tests with this in mind. There's already enough unknown in the tests to add possible earlier failing assertions.

View on HN

Interesting, it sounds like you would benefit from breaking each test out into one that focuses on a single target state.

Another layer would be conditional test execution hierarchy where one early failure would skip a set of tests that depend on that state.

I agree, but the tests are in the order of minutes (sometimes hours), not your typical unit tests. That's basically why it uses soft assertions, it's costly to run. Ideally you'd still want tests with a single assertion/target state, but it's hard to write it that way.

We also do have this conditional test hierarchy, but again it's hard to properly define when the system under test behaves unexpectedly...

> it's hard to properly define when the system under test behaves unexpectedly...

I feel your pain. I think at that point it's good to look deeper and ask if exceptions are being properly thrown, as I try to call for in the final section on Exceptions.

You can't plan for everything.

Seems like the obvious solution is to use hard assertions unless you have a good reason not to.

Indeed that's what I try to advocate. Of course it's not always easy to know if something should be a hard assertion or not until you hit a failure, which is where I think some framework support to run it in a 'soft-assertion-by-default' way would be handy.

Reply on news.ycombinator.com