Hacker News Reader

> Extremely well written and maintained and high quality as of now and having a plan to make sure that can continue in the future are sometimes entirely different things with needs that oppose each other.

I think in this area SQLite has most other open source software beat. SQLite is used in the Airbus A350 and has support contracts for the life of the airframe.

https://mobile.twitter.com/copiousfreetime/status/6758345433...

Other places I have see that they have support contracts through 2050.

This is far more future support than any open source software or likely most commercial software.

Fair points, although the bus factor is more like 3 or 4 for SQLite as far as I know. The question though is that if the entire team vanished from the face of the earth, what would the impact be? My guess is that either SQLite would be good enough as is for 99% of use cases and it wouldn't need much development apart from maybe some minor platform specific use cases or, if new functionality truly is needed, then it would be better for a new team to rewrite a similar program from scratch using SQLite as more of a POC than as a blueprint.

I was speaking less to the SQLite situation specifically and more to the general idea of "Could it be that this is precisely because they don't use "modern tools" and accept outside contributions?" and how I think teams that are very small and not very accepting of outside help/influence might affect that.

To that end I purposefully compares extremes, and tried to allude to the fact that most situations fall between those extremes in some way. SQLite is more towards one end than the other, but it's obviously not a single developer releasing binaries to the world, which is about as far to that extreme as you can go. The other end would probably be something like Debian.

That's not to say either of those situations have to be horrible at what the other excels at. That singular person could have set things in place such that all their code and the history of it gets released on their death, and Debian obviously has a working process for releasing a high quality distribution.

AFAIUI the company behind SQLite (Hipp & co.) have basically endless funding. Not unlimited, just a good enough budget and not likely to end soon. That's also a big factor.

SQLite is supported until 2050, and will likely outlast many other platforms if this goal is attained.

I hope the bus factor is high enough to reach the goal.

"Every machine-code branch instruction is tested in both directions. Multiple times. On multiple platforms and with multiple compilers. This helps make the code robust for future migrations. The intense testing also means that new developers can make experimental enhancements to SQLite and, assuming legacy tests all pass, be reasonably sure that the enhancement does not break legacy."

https://www.sqlite.org/lts.html

> It may not follow that the same can be said a few years from now, or even a few months from now, since the bus factor of that project is one, and if "bus events" includes "I don't want to work on that anymore but nobody else knows it well at all" then for some users that's a problem (and for others not so much).

You may have been speaking generally, and you'd be right, but specifically the bus factor of the SQLite team and the SQLite Consortium is larger than 1, and they could hire more team members if need be.

If and when the SQLite team is no longer able to keep the project moving forwards, then I think we'd see one or more forks or rewrites or competitors take over SQLite's market share.

Yes, I was speaking generally. Specific development models have advantages and disadvantages, but those can often be countered by non-development model actions taken to limit those disadvantages. For example, and extremely open development model is likely prone to more bugs and quality problems, as well as a harder to read and work in code base. There are steps to combat that, such as style guides and automatic style converters, numerous reviewers that can go through code to fund bugs and make suggestions for better quality, etc.

It's not so much that one model over the other will have those problems I've mentioned for each, as much as I think those are common things those projects should be cognizant of and take steps to combat.

As you noted elsewhere, it sounds like SQLite has done a lot that mitigates what I see as the inherent disadvantages of their development model, which is laudable. At the same time I doubt the average SQLite developer is as easily and quickly replaced as the average Linux kernel contributor, even if there are specific kernel developers which would be hard to deal with their loss. Sometimes all you can do is mitigate the harm of a problem, not remove it entirely.

I dunno, I've seen new team members jump onto projects and thrive where most other members have a decade or two of experience. Once you've been around the block dealing with large and complex codebases, picking up a new one gets easier, so I'm not at all worried about the SQLite bus factor. I agree that much larger projects like Linux can have much larger communities to draw leadership from, but I think SQLite is fine.

Once upon a time I wanted to make a contribution to SQLite, and I tried to negotiate making it, but it was quite an uphill battle. On the other hand, I found the codebase itself quite approachable.

> Extremely well written and maintained and high quality as of now and having a plan to make sure that can continue in the future are sometimes entirely different things with needs that oppose each other.

SQLite has both.

And so does the Linux kernel. There are numerous cases of successes and failures at both ends of that spectrum.

My point wasn't to imply that you can only pick one, but that in some cases choices to maximize one aspect can negatively affect the other if care is not taken, and depending on audience high quality released software is not the only thing under consideration in open source projects. Keeping the developer group small and being extremely selective of what outside code or ideas are allowed might bring benefits in quality, but if not carefully considered could yield long term problems that ultimately harm a project.

> > > Extremely well written and maintained and high quality as of now and having a plan to make sure that can continue in the future are sometimes entirely different things with needs that oppose each other.

> [...], but if not carefully considered could yield long term problems that ultimately harm a project.

SQLite has a successful consortium and a successful business model, namely: leveraging a proprietary test suite to keep a monopoly on SQLite development that then drives consortium membership and, therefore, consortium success, which then funds the SQLite team.

This has worked for a long time now. It will not work forever. It should work for at least the foreseeable future. If it fails it will be either because a fork somehow overcomes the lack of access to the SQLite proprietary test suite, or because a competitor written in a better programming language arises and quickly gains momentum and usage, and/or because the SQLite Consortium members abandon the project.

Very good points. The proprietary test suite is clearly the (open) secret to SQLite's success. It seems to me that it isn't even entirely accurate to describe SQLite as written in C when the vast majority of its code is probably written in TCL that none of us have seen. It's more like C is just how they represent the virtual machine which is described and specified by its tests. The virtual machine exists outside of any particular programming language but C is the most convenient implementation language to meet their cross platform distribution goals.

If someone did want to carve into SQLite's embedded db monopoly, it would take years to develop a comparable test suite. This seems possible, particularly if they develop a more expressive language for expressing the solutions to the types of problems that we use SQLite for. Who would fund this work though when SQLite works as well as it does?

MongoDB folks started working on the SQLite alternative - Realm.

Ultimately, the long term harm I was thinking of (for the most part) was lots of proprietary knowledge being lost as a developer is lost for one reason or another, and a resulting loss in quality and/or momentum in the project as that developer may represent a large percentage of project development capacity.

That a large chunk of this knowledge appears to have been offloaded into a test suite is good, and does a lot to combat this, but obviously nothing is quite as good as experience and skill and knowledge about the specifics of the code in question.

As a theoretical situation, how much more likely is a fork to eventually succeed if one or more of the code SQLite developers is no longer available to contribute to SQLite? There are a lot of variables that go into that, but I would feel comfortable saying "more likely than if those developers were still present". That idea encapsulates some of the harm I was thinking of.

Institutional knowledge, and leadership, is indeed critical. The knowledge of a codebase can be re-bootstrapped, and its future can be re-conceived, but actually providing leadership is another story. I think there's one person on the SQLite team besides D. R. Hipp who can provide that leadership, but I'm not sure about business leadership, though who am I to speculate, when I don't really know any of them. All I can say is that from outside looking in, SQLite looks pretty solid, and libSQL seems unfunded.

> Extremely well written and maintained and high quality as of now ...

As I see it, the quality comes from the fixed need, tight scope and small professional team

Fixed need -> Somebody always needs a database

Smaller scope -> Less features, less code, less bugs

Professional team -> No newbies around to break things

Now, to be sure, this kind of development model can only work for some projects.

Also tooling focused on their dev flow.

In difference to the article implying that they use out-date project tooling they don't(). They VCS isn't out-date but in some way more modern then git, it's just focused on dev flows similar to theirs to a point where using typical git dev flows will not work well. Similar not using GitHub is a more then right decisions for how the project is manage, github is too much focused on open contribution projects.

(): You could argue they use "not modern" tools like C and similar to do modern things like fuzz testing. But the articles author clearly puts a focus on the project tooling highlighting git/GitHub so reading it as implying that their VCS is "not modern" i.e. outdated i.e. bad seem very reasonable IMHO.

> Extremely well written and maintained and high quality as of now and having a plan to make sure that can continue in the future are sometimes entirely different things with needs that oppose each other.

Software that's truly "extremely well written and maintained and high quality as of now" has the option of a plan like:

> "At the time of my death, it is my intention that the then-current versions of TEX and METAFONT be forever left unchanged, except that the final version numbers to be reported in the “banner” lines of the programs should become [pi and e] respectively. From that moment on, all “bugs” will be permanent “features.” (http://www.ntg.nl/maps/05/34.pdf)

If your software needs perpetual maintenance, that's a good sign that it's probably not that high quality.

> If your software needs perpetual maintenance, that's a good sign that it's probably not that high quality

The problem in a lot of cases is not the software per se, but changing environments. Windows upholds backwards compatibility to a ridiculous degree (you can still run a lot of Win95 era games or business software on Win10), macOS tends to do major overhauls of subsystems every five-ish years that require sometimes substantial changes (e.g. they completely killed off 32-bit app support), but the Linux space is hell.

Anything that needs special kernel modules has no guarantees at all unless the driver is mainlined into the official kernel tree (which can be ridiculously hard to achieve). Userspace is bleh (if you're willing to stick to CLI and statically linking everything sans libc) to horrible (for anything involving GUI or heaven forbid games, or when linking to other libraries dynamically).

The worst of all offenders however is the entire NodeJS environment. The words "backwards compatibility" simply do not exist in that world, so if you want even a chance at keeping up with security updates you have an awful lot of churn work simply because stuff breaks left and right at each "npm update".

You say nothing seriously false, but perhaps depending on things like NodeJS just inherently means your software is going to be poor quality. If that is true, then both you and the PP are probably right. I tend to think software quality will be higher if you depend on a third-party collection (such as so-called Linux distributions) than a second-party aggregation (such as NPM).

Even the distributions have a hard time with the NodeJS environment and its relentless pace - and the more software gets written in JS the worse. When e.g. software A depends on library X@1.0 and software B on library X@1.1, and X has done breaking changes, what should a distribution do?

Hard forks in the package name (e.g. libnodejs-x-1.0 and libnodejs-x-1.1) are one option, but blow up the repository package count and introduce maintenance liability for the 1.0 version. Manually patching A to adapt to the changes in X works, but is a hell of a lot of work and not always possible (e.g. with too radical changes), not to mention if the work should be upstreamed, then licensing issues or code quality crop up easily which means yet more work. Dropping either A or B also works, but users will complain. And finally, vendoring in dependencies works also, but wastes a lot of disk space and risks security issues going unpatched.

And that's just for final software packages. Dependency trees of six or ten levels deep and final counts in the five digits are no rarity in an average NodeJS application.

Importing even the bare minimum introduces an awful lot of work and responsibility to distributions.