Joran from the TigerBeetle team here.
This was in fact one of our motivations for static allocation—thinking about how best to handle overload from the network, while remaining stable. The Google SRE book has a great chapter on this called "Handling Overload" and this had an impact on us. We were thinking, well, how do we get this right for a database?
We also wanted to make explicit what is often implicit, so that the operator has a clear sense of how to provision their API layer around TigerBeetle.
Being crash tolerant is one thing. But "crash early, crash often" is absolutely horrible advice.
Sure, but again, that's a scenario that you need to handle anyway.
> Being crash tolerant is one thing. But "crash early, crash often" is absolutely horrible advice.
I've found it to be good advice. It's a big part of why Erlang systems have been so reliable for decades.
And that's why Erlang is so resilient, precisely because the semantics of the language make it easier to isolate tasks and subtasks, minimizing blast radius and the risk of reaching an inconsistent or unrecoverable state. I often use Lua (as a high-level glue language) to accomplish something similar: I can run a task on a coroutine, and if a constraint or assertion throws an error (exceptions in Lua are idiomatically used sparingly, similar to how they're used in Go), it's much easier to recover. This is true even for malloc failures, which can be caught at the same boundaries (pcall, coroutine.resume, etc) as any other error.[1] It's also common to use multiple separate Lua VM states in the same kernel process, communicating using sockets, data-copying channels, etc, achieving isolation behaviors even closer to what Erlang provides, along with the potential performance costs.
[1] While more tricky in a language like Lua, as long as your steady-state--i.e. event loop, etc--is free of dynamic allocations, then malloc failure is trivially recoverable. The Lua authors are careful to document which interfaces and operations might allocate, and to keep a core set of operations allocation free. Of course, you still need to design your own components likewise, and ideally such that allocations for connection request, subtasks, etc can be front-loaded (RAII style), or if not then isolated behind convenient recovery points. Erlang makes much of this discipline perfunctory.
The situation you described is exactly what is prevented with static allocation. It's a very predictable design - it will successfully handle X requests and will fail any requests above that.
An important thing implicit in this design - or at least that I'm assuming - is that there is no queue of requests. Queuing is a major source of problematic behaviour during overload. It's /queuing/ that causes RAM to increase with load (in a system with fixed max of requests being processed). It's /queuing/ that is why latency increase as you near load limits (vs errors).
I'm with you that early errors are better, and the scheme used here achieves that on a request level.