Hacker News Reader

Comment by TAForObvReasons | Hacker News Reader

There was an interesting comment a few months ago [1]

> Not mentioned is that the full test sqlite test suite is proprietary and you need a super expensive sqlite foundation membership to get access to it.

According to Dr Hipp [2], no one bought the test suite. So there are definitely deficiencies in the test suite which may have been better addressed if the full test suite was open.

[1] https://news.ycombinator.com/item?id=33346661

[2] https://corecursive.com/066-sqlite-with-richard-hipp/#billio...

View on HN

From the second link:

> We still maintain the first one, the TCL tests. They’re still maintained. They’re still out there in the public. They’re part of the source tree. Anybody can download the source code and run my test and run all those. They don’t provide 100% test coverage but they do test all the features very thoroughly. The 100% MCD tests, that’s called TH3. That’s proprietary. I had the idea that we would sell those tests to avionics manufacturers and make money that way. We’ve sold exactly zero copies of that so that didn’t really work out. It did work out really well for us in that it keeps our product really solid and it enables us to turn around new features and new bug fixes very fast.

It also makes forking SQLite infeasible since any new changes will be woefully under tested.

SQLite's free test suite is far in excess of most other projects'. What do you think the test coverage is in the Linux kernel, GCC, or Emacs?

(Embedded) database has stricter requirements than a text editor or a compiler.

Obviously not, if nobody was willing to buy the test suite for their internal forks.

That's a non-sequitur argument.

It follows fine, but you want to debate instead of think.

If the 100% MC/DC coverage was critical to forks, the companies that fork (there's lots of them!) would have bought it.

Nobody bought it, so it's not that important to maintaining a fork compared to the regular test suite even for such environments. A test suite which, to go back to my first comment, is still leagues ahead of the dozen other pieces of lynchpin software most companies have no problem depending on.

Meanwhile, for the 99.9% of us out here not building aircraft and merely shipping a billion browsers or phones...

> If the 100% MC/DC coverage was critical to forks, the companies that fork (there's lots of them!) would have bought it.

Again, that's a non-sequitur (or perhaps strawman, you can choose), because I wasn't addressing the proprietary test set, merely the comparison between a text editor and a database, which is completely absurd since the tolerance for failures is drastically different.

> I wasn't addressing the proprietary test set

Then perhaps you're in the wrong thread to be saying anything at all.

Right, but it makes the fork strictly worse from a reliability perspective than SQLite given it will be less tested.

If there wasn't a competitive advantage given it has no sales, wouldn't they have open sourced it by now?

> wouldn't they have open sourced it by now?

If there's minimal value in it, why put in the work to open source an extremely complex test environment?

Just for completeness sake, they do offer a SQLite Consortium Membership for $120k, which I guess includes all their test suites as a selling point: https://www.sqlite.org/prosupport.html

What was the incentive to buy these tests if they've already been used to improve the product?

You can then audit the tests to ensure they test some condition that you are concerned about.

In reality nobody audits source code like that (see heartbleed for an unrelated example of critical code that didn't get proper audits from people who should have cared)

But the testsuite being proprietary is part of their business model.

Reply on news.ycombinator.com