Otherwise, the same story could be told as a generic software testing joke: "unit-tests are short-sighted and coverage lies", i.e. an "extremely well-maintained codebase, with extensive unittest, >98% test coverage and constantly scanned by all-static-analyzers-you-may-come-up" can have fatal, trivial bugs.
I've never understood that to mean that he wasn't in favor of automated testing, only that it's got its limits. In this case, they now know a test case that was missing.
Is that special? Are there buffer overflow bugs that are not security bugs? It could be just my bubble as a security consultant, since (to me) "buffer overflow" assumes remote code execution is a given. It's not my area of expertise, though, so perhaps indeed not all reachable buffer overflows are security issues. (Presuming no mitigations, of course, since those are separate from the bug itself.)
https://googleprojectzero.blogspot.com/2014/08/the-poisoned-...
Not necessarily:
1. Most compilers use aligned variables and fields unless forced not to with a flag. Going 3 bytes over an array of 12 bytes can result in an overflow that is never detectable at runtime because the extra memory being used is used exclusively by the bug.
2. Malloced memory is harder (but not impossible) to turn into an unintended execution because the pages may be marked for data only by the OS. The stack pages are ~marked~ [EDIT: ~NOT marked~] as executable.
There's probably millions of buffer overflows that will not only never be exploited, but will also never be detected.
If all you have is a stack overflow you may have your work cut out for you.
Further, while this bug may exist, assertions elsewhere could have meant it was unexploitable. So in isolation it's an overflow but in context it's not reachable. That didn't happen here. This happens a lot actually.
They enable refactoring code in ways not possible without them.
Granted, even tests that have to be changed have some value in securing a refactoring.
Once those tests have been written, then they act as regression tests like the parent comment notes.
On "unit tests", I view the behaviour of the class/function and all its dependencies as a single unit for the purpose of testing. I've never liked the idea of needing to mock out a class in order to test another class, just because the class being tested makes use of it. The only case where mocks/stubs/etc. should be used is when interacting with an external component like a database or HTTP API. -- You don't see people that do this mocking out a list, string or other library classes, so why should project classes be any different.
I agree that there is no reason to mock data containers or other classes with fully self-contained behaviour.