Java Mutation Testing
pitest.org
pitest.org
Thanks for sharing. I will play with it today!
Relevant plugin for IntelliJ: https://plugins.jetbrains.com/plugin/7119?pr=idea
Because end-to-end integration tests don't actually assert anything about model properties, it's incorrect to record model coverage during such tests. So each controller test only recorded coverage for the controller it was testing (and nothing else), each model test only recorded coverage for the model it was testing (and nothing else), etc.
Of course, in a more IoC/DI-type system (Spring, etc.) well-written tests don't interact with other objects in the first place: stubs or mocks are injected. In that case it's easier to ensure a test for a given component only exercises that one component, but you still have to verify that your assertions are meaningful for the coverage to ultimately mean anything.
So I guess what I'm saying is even with all those precautions and thought, a tool like this is extremely useful to tell you "uh, hey, this method you think exercises all your code isn't actually asserting anything meaningful about your code's behavior". More of this, please!
For instance, changing the constant to another prime in the "classic" hashcode implementation (repeatedly multiply by a prime and add the next field) will (probably) not trigger any (well-written) tests, and indeed generally won't be detrimental at all, but will be flagged by this sort of test.
(By the way, I think you're overstating your case when you claim "there are plenty of places" where it doesn't matter. I think in the vast majority of cases, if you change a "<" to a ">", you introduce a bug!)
The problem I have is this: this testing framework explicitly states the following:
> The quality of your tests can be gauged from the percentage of mutations killed
So, you write your tests to test that there is a consistent ordering, it changes the "<" to a ">", finds that all tests still pass (because, as you said, you're only testing if there is a consistent ordering, because that's the only thing that matters in this case), and considers your tests lower-quality because of that.
This is a flaw most mutation testing - and, for that matter, test driven design - suffers.
For another flaw: there are many things that end up along the lines of a bunch of shortcut "fast paths" at the start, with a slow path at the end that does slow but exhaustive checking.
For example, if you're checking that two objects are equal you may insert a check at the start to see if they are the same pointer, and if so return `true`. If you're doing division in software you may check if you're dividing by 1 instead of doing the computation. That sort of thing.
Any mutation that introduces false negatives into the shortcuts (but no false positives) won't be flagged by mutation testing.
Along with high computational cost, it's one of the things identified in academic research as preventing the widespread use of mutation testing. There is no general method for distinguishing an equivalent mutation from a normal surviving one except for getting a human to have a look at it.
Having built pitest to address the concerns around computational cost, it was a pleasant surprise to find out that in practice equivalent mutations are not much of a problem.
This isn't entirely by accident - the default set of mutation operations are carefully designed to make equivalent mutations unlikely (they don't/can't guarantee not to create them, but they make them as unlikely as possible).
There's a trade off here. Pitest has a smaller set of operators than a lot of research focussed systems. A larger set of operators would catch more issues, but would also create a larger proportion of equivalent mutants (and also take longer to run).
There are more operators you can enable you wish to change this balance - an operator that changes constants as you describe is one of them.
I rarely encounter equivalent mutants using the default operators and I know of some rollouts of pitest where they break the build on anything less than 100% mutation coverage.
I have no figures to back this up, but I strongly suspect the % of equivalent mutants will be highly dependent on coding style and the domain in which the code operates.
Agreed. In everything I've written, at least, I can identify multiple places where such equivalent mutants do exist, even with just the default operators. (In particular, hashcode methods - there are very few mutators that break the contract of a hashcode method, though removing entropy in most cases) But I can easily see that not being the case with other coding styles.
Are the mutations done documented anywhere? I had to look through the source to see what's done.
Unfortunately, we had to remove it from our build. Our CI pipeline uses VM's that were not provisioned with this type of testing in mind and pitest ended up slowing the build enough to make it painful. If we could get past this, I would turn it on in the morning.
Or even simpler, why not have build actions copy it to/from a specific location outside the build target dir? You could even make it a shared network drive, if you didn't care about the possibility of losing someone's changes if two people run tests at the same time.
I've never used one of these tools, but I knew they existed. Just today I was discussing "testing the tests" with a coworker. In my opinion, where I work we write lots of incomplete/illogical tests, sometimes bordering on cargo culting. Aside from code-coverage, which is flawed, we have no real measure of whether the tests we write are effective or not.
For other kinds of test quality, try Mutator: http://ortask.com/mutator/
I hate to sound negative, but something about that website seems dodgy. I'm unconvinced duplicate/overlapping tests -- while obviously undesirable -- have a direct correlation with code quality. Unfortunately, in order to read their "papers" where they elaborate on this, I have to register :/
I'd rather use something open source like pitest.
(Again, thanks for the links! I don't want to sound too negative)
Does anyone know if the current version of pit already works well with scala?
if you are using interfaces to inject and mock things, it cannot test your code?
this sounds really cool, but most of the stuff we do is down with interfaces...
You may mock dependencies that may/may not be interfaces. This is safer than using concrete dependencies whose behaviors may change once the fuzzer does its thing.
The behaviors of the classes that implement the mocked dependencies have nothing to do with the system under test.
So if you're writing unit tests that exactly one unit, you should be good to go.
Namely, what happens when you're done with the test-code-repeat cycle for a system under test and you want to make it more architecturally sound / OO / etc.
You typically might end up doing a refactor in which you extract classes from the original system under test and colocating common functionality into new class(es).
In this way you still have the same coverage as before... But you're actually testing multiple classes as a unit.
Some folks would argue you need to split the tests out. Otherwise would say "the coverage is there, what's the point?".
Unless I need to do something, I'm not going to do it.
In playing with pitest, it looks like the refactorings might introduce some fuzzing that needs to be considered in the original (and refactored) SUT depending on how much conditional logic you're moving around.
Arghh. There goes my evening. Will be playing with this after work now :)
The sentence you quoted relates to an optional feature that allows the tests that will be run against a mutated class to be limited to those within a certain "distance" (i.e number of method calls) from it.
The feature is little used, but is useful in some very specific circumstances.
Even if you do enable it, it would only cause a problem if the class under test was only referred to within the test by some interface that it implemented (which would be very unusual). The fact that the classes dependencies were declared as interfaces would cause no problems.
Its fast, ...
with
It's fast,...