What are you testing for?
This is critical because it basically gives you immediately what you should and should not test, and how. While mindless, dogmatic, metric oriented testing is a waste, testing with higher intent and purpose is extremely useful.
An example: test that something working on current vX also works on vA to vW, and when vZ is out, have the answer readily. Or that a biz feature fulfills the requirements. Or that someone not as well versed on intricate details of your piece of ownership will be confident in that piece still working after a simple fix when you’re on vacation. It can be one, some, but probably not all.
With that in mind, what to test, what doesn’t make sense to test, and what to test against becomes more clear: should I mock this? or should I run it against some staging environment? Should I perform (yikes? not!) manual testing?
The answers are highly dependent on the piece of code being tested.
Tests are here to help you answer a question, if you aren’t sure what the question is then your tests will miss the point.
100% coverage of what exactly? Tests that go through all your lines of code without testing any of the logic, is useless. If you want to be thorough, you need to do mutation testing, which is a system that tests the quality your unit tests by mutating your logic (changing a > for >=, a + for -, etc) and then expects at least one test to fail. If no test fails, that piece of logic wasn't tested.
Without that, it's entirely possible your high code coverage doesn't actually test anything meaningful. Also, this sort of logic is exactly the kind of stuff you want to unit test. All the standard plumbing boilerplate code is not something that needs to be unit tested. The logic does.
I’d also like to add that if you contribute code to an open source project it is extremely beneficial to have iron-clad unit tests. Since there is so many devs it would be easy for someone to accidentally break something you fixed already.
It's often easier to just aim for 100% test coverage instead (with excluding some categories of files).
EDIT: I would not and did not start with 100% unit testing. But if there are ongoing culture wars and discussions didn't lead to a workable compromise, 100% test coverage worked for me and after some days test coverage was a non issue.
Property based testing and random values in unit tests find lots of bugs you didn't think of though.
Rather, you won't find bugs that you choose not to think of because you've let the "100%" number lull you into complacency, even though you know it's 100% of lines/branches, not 100% of inputs.
My problem in 40y of programming is still making bugs and those I make come from not thinking about edge cases or from wrong assumptions and not from being lulled into writing tests to meet a 100% number.
But personalities differ and if being lulled into security by writing towards a 100% number is a problem for you I would be careful, I totally agree here.
Perhaps I am wrong, and I would not start with a 'diktat' for obvious reasons.
As a manager, you didn't have discussions about the level of necessary code coverage? Would be interested on how you managed unit testing without 'dictat'. How it would fit into integration testing and explorative testing. What level did developers in your deparment usually find "adequat" ? If you considered it too low, how did you raise test coverage as a manager without defining a coverage level?
But in this case I think the cure might be worse than the disease. Tests for plumbing code often end up being brittle tests of methods getting called on mocks in the right order. People will notice that these require a lot of toil to keep them running as code changes while providing very little benefit in avoiding mistakes. People will rankle at being told that they must write these tests, which they can see are a waste of time.
I've done it both ways. I'm much happier with my work when I'm not trying to write tests that are tedious and don't seem to provide any value, in order to hit an arbitrary coverage metric. I suspect my teammates feel the same way, so on teams where I have input into the decision on this, I do not advocate for 100% coverage. It does make it harder to have the discussion of which tests should and shouldn't be written, but I think it's worth that cost.
Writing good testing code is harder than writing business code. Especially junior developers struggle with this, most often because many companies write not enough tests to learn writing good tests.
And if you're in an environment, where this is a non-issue I think thats great. Don't fix something that doesn't need to be fixed.
It has a side benefit that it forces devs to write testable code, which inclines them to reasonbly factored code.
I'd prefer <100% coverage plus discussions about what to test (and how) much more than working with a test suite built on the wrong incentives.
That's where the 'gaming' comes in.
The tests start just going through lines without hitting a single expect statement.
The ignore files start becoming battlegrounds in the PRs because people just exclude half the damn project.
We just have a simple rule... if you wrote code, you have to write coverage for it. If it breaks and your test doesn't catch the breakage, the bug fix goes back to you. Some people will ask "but what about what I'm working on now", you'll have to communicate that you feel your previous work was far more important.
this feels punitive, especially in the eyes of management. unless you're in a safety critical area where fully testing every code path is a hard requirement, people will eventually write bugs.
i'd rather work somewhere that recognizes defects occur and has a fast iterative process to push out new changes rather than one based on shame for having written a bug.
That is the fastest most iterative process we have found so far... as the expert on the original code, you are able to deliver the best outcome.
You're not being shamed for writing a bug, you're being shamed for not testing your code.
When my colleagues are knowledgable and open minded I would embrace every opportunity to have a good discussion.
Congratulations, now you have a war over which categories of files are excluded from the "100% test coverage" rule. ;)
I strive for working code. Sometimes I miss something in the TDD cycle and don’t have 100% and it is that which usually comes back to bite you.
I have never found 100% test coverage has bitten me, dogmatic or otherwise.
switch(type) {
case X:
...
case Y:
...
default:
throw InternalException("Unsupported type!");
}
switch(type) {
case X:
...
default:
assert(type == Y);
...
}
Is worse code worth getting 100% code coverage? In my eyes, absolutely not. I think good code + testing should be able to reach at least 90% typically, likely 95%, but 100% is often not possible without artificially forcing it and messing up your code and/or making it much harder to change your code later on.
You can be defensive to various degrees about assertions:
1. You can just use assert() to fail in Debug and do nothing in Release. 2. You can be more defensive and define your always_assert() to fail in Release as well. 3. You can double down on the UB with hints to the compiler and provide assume(), which explicitly compiles to UB when it's triggered in Release (using __builtin_unreachable() for example).
About the organization of the if statement: I agree that the former is better, I would use assert(false) though.
Throwing an exception here is basically free (just another switch case) and gives the user a semi-descriptive error message. When they then report that error message I can immediately find out what went wrong. Contrasting with a report about a segfault (with maybe a stacktrace), the former is significantly easier to debug and reason about.
assert_always would provide a similar report, of course. However, as we are writing a library, crashing is much worse than throwing an internal error. At worst an internal error means our library is no longer usable, whereas a crash means the host program goes down with it.
Better yet, omit that default case, so that in the future when you do add a new value to the enum, the compiler will warn you and force you to add a new case.
But I agree with your general thesis that it's just not worth getting to 100% coverage.
Tell that to SQLite guy.
Try that with a networked application that takes user input though...
It will always break on the user's internet, because it's too diverse to predict.
Doesn't mean you can't have some networking unit tests, just that you shouldn't believe in them too much.
Edit: you said services. You thinking of server? I'm thinking of clients.
And it doesn't have to be a static mock. It's not too hard to inject a fuzzer in your mock service response, although that's probably left to a separate testing routine, and not part of your unit test setup. But if you have no mock for your network service, you can't fuzz it either.
SQLite is suitable for 100% coverage.
A lot of application code or workflow style code is hard to reach 100% coverage as they are rarely triggered.
That's an opportunity to describe, in code, what that path is supposed to do, and then make sure it does it.
Which shows 100% unit test coverage is not better than spending that time in other kinds of tests.
SELECT code execution from using SQlite - DEF CON 27 Conference
https://www.youtube.com/watch?v=HRbwkpnV1Rw
Don't get me wrong, it's a great product and I use it often, but 100% test coverage does NOT equal 100% safe.