If you find this many critical bugs, there is no doubt that there are more, since it is indicative of the overall code quality. Still, what they're doing is good work and crypto remains hard. I am glad they're doing it.
This was a third-party audit, and VeraCrypt fixed all of the critical ones that they found. All software has bugs. To find some that are critical is not necessarily indicative of the quality of code on the whole.
auditing a large codebase certainly isn't as easy as proofreading a paper. it's not a "one and done" scenario. especially when you also factor in the security requirements of encryption software. the world won't end if there's a missed spelling mistake or two in a term paper, but if there are critical vulns in VeraCrypt the entire software could be rendered useless
Smoke, fire, yadda yadda.
Critically dangerous, but understandable(complex crypto/ interactions). Those do not indicate low code quality.
Critically stupid/unwise(you know what these are, e.g. the libressl rant list). These do.
This is a form of "slippery slope" argument that holds no weight as to the security of VeraCrypt, one way or another.
Between two pieces of software with the same function, if the same audit process finds 10 critical bugs in software A, and 100 critical bugs in software B, which do you think is more likely to have undiscovered bugs?
I'd say B, since no bug discovery process is perfect. For example, assuming a ~90% hit rate, you'd expect there is ~1 undiscovered bug in A, and ~10 in B.
(Note that I am not saying VeraCrypt is bad or that you shouldn't use it. I have no idea how it compares to similar software. All I'm arguing is that, ceteris paribus, discovering more critical bugs is at least weak evidence that more undiscovered critical bugs exist, since no bug-discovery process is perfect.)
That's the argument being made implicitly, and the argument you're reiterating.
Saying, "Who knows if there are more bugs?" is a useless statement, you simply can't do anything with it. That's the nature of bugs, if you can't identify or find them, you can't claim they exist.
100 bugs in software B does not imply the 101st bug, is my point. There's no such thing as a bug "hit rate".
I did not make that argument, I made a probabilistic argument. Trying to apply deductive logic fallacies to probabilistic arguments is a type error.
>if you can't identify or find them, you can't claim they exist.
I made no absolute claim about the existence or nonexistence of bugs, only about the expected number of bugs, that is, a probabilistic estimate.
> There's no such thing as a bug "hit rate".
Of course there is, it is, trivially, the number of bugs found by the audit / the total number of bugs.
Look at this way: I have two auditors, Alice and Charles. Charles is known to be pretty sloppy, he looks over code quickly, skip parts, etc, but the bugs he finds are usually actual bugs. Alice is extremely rigorous and misses very few bugs.
I have 2 pieces of software, SuperCrypt and UltraCrypt.
Charles audits SuperCrypt and finds 137 critical bugs. He audits UltraCrypt and finds only one.
I know Alice will review both code bases next week. Where do you think she'll find more bugs?
You did make the argument I presented above, even if you didn't mean to. It is not a type error, it is a conclusion that you're trying to imply but don't want to directly state because it's indefensible, and you know it.
There is not such a thing as a bug "hit rate" because it would require knowledge of unknown bugs, which isn't possible.
To be abundantly clear, I find your emotional narrative entirely unhelpful to this conversation, but to illustrate how unhelpful, I will play along:
UltraCrypt clearly wasn't audited correctly, and will have more bugs in it given the complexity of the problem space and the lack of issues found. SuperCrypt was clearly audited well and will therefore have fewer bugs if Charles continued looking and when Alice looks next week.
This, I repeat, is wholly unhelpful, because the same information can be used to create arguments for either point, and is therefore not useful.
I often try to back up abstract arguments with a concrete illustration, it makes things clearer for me.
I've clearly failed to make myself understood, I don't think it's useful to continue this conversation.
Suppose you're a professor and you had a super-productive student who wrote 100 very interesting papers. You only have time to read 50 of them, selected at random. However, you notice that in 10 of these 50 papers, there are mathematical errors. What do you expect of the remaining papers, how many errors will there be? This is basic statistics and nothing about this argument is emotional.
What if you read 50 papers and only one contained mistakes? Would you expect more or less errors now in the remaining papers?
It is possible to estimate the number of things you have not seen. The German Tank Problem [0] is a famous example. For software, if you Alice and Bob independently look at a piece of code for bugs, and Bob finds 17, Alice find 21, and 16 overlap, then you have some knowledge of Alice and Bob's rate at finding bugs, and you can estimate the number of bugs expected to be left. As the sample sizes increase you have smaller and smaller confidence intervals, giving quite good knowledge of the count of unknown things.
You are assuming once an audit is done, all bugs are found. This is rarely the case, and I doubt has ever been the case on software as complex/large as this piece.
He is assuming an audit can find bugs, each with some probability. This is a quite reasonable position, one borne out by practice.
His view is true in practice (and theory, since proving software correct is tremendously difficult except). There is ample literature on how many bugs are likely left after an audit, and this audit itself shows that these bugs remained after people (presumably) looked over the code previously.
More effort/skill usually turns up more bugs. It's that simple. Do another audit with more skilled eyes, paid more money, given more time, and they will likely find bugs that this one did not.
There is a large literature on estimating the number of bugs left after some are found. Here [1] is an example of how this is done.
[0] https://en.wikipedia.org/wiki/German_tank_problem [1] http://www.johndcook.com/blog/2010/07/13/lincoln-index/
In fact, your second link begs the question, as it requires the prior knowledge of a "bug hit rate", which is defined in the blog post as a fairly arbitrary number, with plenty of caveats, and requires multiple independent audits.
In fact, from your linked blog, "Maybe the Lincoln estimate is not at all accurate, but it does tell you to be worried that there may be a lot more bugs to find since the overlap between the two bug lists was so small."
I begin to wonder if you actually read your own links...
"There’s a simple statistic called the Lincoln Index that lets you estimate the total number of errors based on the number of errors found"
"Suppose two testers independently search for bugs. Let E1 be the number of errors the first tester finds and E2 the number of errors the second tester finds. Let S be the number of errors both testers find. The Lincoln Index estimates the total number of errors as E1 E2/S. "
E1 = count from searcher 1 E2 = count from searcher 2 S = Number overlapping
If S != 0, then a good estimate of the number of bugs in the code is E1E2/S. One can derive error bounds and confidence intervals if desired
Nowhere does this formula need prior knowledge. All is needs is at least two testers to count the bugs each found and the overlaps. This is precisely* the example I gave above.
As one analyzes how this works, one finds that this is likely lower than the true bug count. A simple reason is explained in "Beginning Software Engineering" by Rod Stephens, p197, which you can find on google books and read.
Alternatively, try searching for "estimate number of software bugs" in Google scholar and you can see a vast literature based exactly on refining methods like this.
I never said the Lincoln Index required prior knowledge, I said the German tank problem required prior knowledge that isn't available in the VeraCrypt security audit.
Your example may have been copied from the blog post, but you failed to include any of the caveats that were present in the blog post, which are also present in the "vast literature" in Google Scholar.
Rod Stephens even says, in the very citation you attempt to make, "Like all the other bug estimation techniques, this one isn’t perfect. It relies on the assumption that the testers have an equal chance to find any particular bug, and that’s probably not true." It does go on to say the Lincon Index, "probably underestimates the true number of bugs." but in that aspect, Rod Stephens disagrees with the body of work available both in your own links, and in the Google Scholar articles. He then cites Wikipedia, so in that he's basically admitting he's not an expert.
I am not and have not been saying there are no further bugs in VeraCrypt.
I am saying the likelihood of there being "many" more bugs in VeraCrypt is not available, given the data we currently have. We don't have E2, for example, since you're focused so intently on the Lincoln Index.
You wrote
>In fact, your second link begs the question, as it requires the prior knowledge of a "bug hit rate",
so you wrote exactly that the Lincoln index "requires the prior knowledge of a bug hit rate".
>which are also present in the "vast literature" in Google
Ah, so you read all the literature. Good. Then you're realize there are other methods than Lincoln, so your complaint about not knowing E2 is mitigated.
>I am not and have not been saying there are no further bugs in VeraCrypt.
You wrote above
> If there were more bugs, why weren't they found?
What did you mean here except to imply you believe there are no more bugs?
>I am saying the likelihood of there being "many" more bugs in VeraCrypt is not available, given the data we currently have.
Here [1] is the report.
It states in multiple places not all code was checked. For example, page 6, states that "Some components of VeraCrypt have not been investigated" including "the OS X and Linux versions of VeraCrypt". Want to bet if there are bugs in there?
Of the parts checked, bugs were found by multiple independent groups, including Open Crypto Audit Project, Fraunhofer Institute for Secure Information Technology, and James Forshaw. These bugs have very little overlap.
OCAP phase I found 11 bugs, page 4. OCAP Phase II found 4, page 16, two of which overlap previously known bugs. James Forshaw found 2 more bugs, page 18. Sections 5, 6, and 7 each list more bugs by (presumably) the two people listed working on this portion.
Given less than 100% code coverage and the lack of independent groups finding the same set of bugs, makes it extremely likely there are more bugs.
This paper itself demonstrated audits by one group followed by later audits by another group, each finding more bugs.
Yet you want people to believe there are no more bugs since we have not located them, despite the vast evidence and literature that the above points to more bugs lurking.
As you stated, "If there were more bugs, why weren't they found?"
[1] https://ostif.org/wp-content/uploads/2016/10/VeraCrypt-Audit...
I think you've misread the report, as there weren't independent groups, it was two people from QuarksLab who performed the audit. Right there on page 6, the page you cite. There weren't multiples of the same bugs found by these two people in the report (why would they both report it?), OCAP phase I and II weren't done on VeraCript, and James Forshaw hasn't looked at VeraCrypt at all. This was E1. There is no E2.
The alternative methods for determining bugs are all as flawed in this context as the one you've weirdly chosen to champion, the Lincoln index. What other specific method would you like to attempt to apply here? I'll gladly walk you through why it won't make sense in this context.
I'll say it again: I never said there were no more bugs. You've built this strawman because it's easy to dismantle, however it isn't real.
What I asked was, as you've quoted, "If there were more bugs, why weren't they found?" This draws an important line between the theoretical and the actual; the accusation of bugginess can be levied at any program. It is of no use or value to anyone to say such vague things as, "VeraCrypt may have more bugs!" Of course it could, that's a worthless statement. What's valuable is actually demonstrating those bugs.
It's painfully clear you didn't actually understand the report, and you quote me twice, despite admitting you didn't understand the quoted text.
I'll repeat here for you: I don't think and never said there are no remaining bugs in VeraCrypt, I do however believe it is useless to essentially exclaim to everyone that VeraCrypt is a piece of software (that's what you're doing when you're saying it might have other bugs).
If there are other bugs, where are they? Surely you can find them if you know they exist.
Odd, since the rest of that sentence discussed a blog post, and the first link is a Wikipedia article, while the second is a blog post. Nothing in the German Tank problem corresponds to prior knowledge of a rate of any kind. The word "rate" does not even appear on the page. It does appear in the blog post.
Your repeated intellectual dishonesty makes further discussion pointless. You are bizarrely resistant to admit there is an entire field called software defect estimation, or that future trends for bugs are not independent with current bugs. I give up like everyone else in your thread.