Hacker News Reader

Comment by gkfasdfasdf | Hacker News Reader

For a great explanation of why you would want to use mutation testing, see https://pitest.org/ (a similar tool but for Java). In short - it is to gauge the quality of your test suite, because ideally any mutation of your app should cause a test failure.

View on HN

Well, not any mutation. Deleting an assertion should not cause any problems. Replacing bubblesort by quicksort shouldn't either. Etc.

Hmm, I’m thinking that removing an assertion would potentially allow your program to continue into an undefined state that the assertion was guarding.

You’d also want to watch out for the particular implementation of sorting algos... if you swap out a stable quicksort for a nonstable bubblesort, your benchmark tests might still pass, but you could wind up with different orderings of “equivalent” items in your list before/after that could cause subtle issues.

If you have a test that truly still passes regardless of any mutation, I’d have to wonder if it should exist at all.

This is a valid concern, for sure. In fact, these are called Equivalent Mutants. We observed it several times, here are some examples. You flip the if/then branches and the code essentially does the same, switching from a fast to slow implementation of the same algorithm. Another example is mutating the code in a C++ destructor, which is not easy to test.

With that being said, even despite the equivalent mutants, you can get a pretty good insight. Here is an example of finding from a proprietory real-time OS used in the space industry: https://gist.github.com/AlexDenisov/b5d2e23457b88813b5ab9d5d... (this is a part of an email which I could safely post online).

Reply on news.ycombinator.com