> Using a text file and ... > It seems like something you'd do as a proof of concept
I mean, using a text file for the proof of concept is exactly what happened. And then it grew too quickly and had to get off of it eventually, but we always knew that. We were just amazed how long that worked. It survived much longer than we'd thought.
This might actually be a really good filter for which types of engineers are good fit for their company. Because my read was very different than yours, as I looked at the decision lineage and thought to myself I could see myself making every one of those decisions. I of course can't say I would've with the same problems and constraints they had, just that I know I've considered similar solutions.
The big difference might be, on the third iteration I tend to land on embedding raft directly into the backend, because lots of the problems I look at tend to benefit from a in-order log that is fully replicated to a quorum of servers. But based on the post and efficiency they're going for backing up the WAL on sqlite may make alot of sense.
Actually litestream looks like it might have some fairly similar properties, so might be something else I'd like to add to my toolbox. Although I wonder what sort of replication lag there might be to replicas when using s3 as the stable store.
A good engineer can build all sorts of surprising things. A senior one knows why they should reserve that for only the special bits. I like reading the first two types of articles -- we frequently give tech talks ourselves -- but for making boring systems weird, I get flashbacks of consultantware/resumeware/NIH that took ages to weed out. I try to give talks on say GPUs for weirdly shaped problems, not on why we celebrated when we finally had time to rip out mongo from where it didn't belong :)
I've worked with Raft a lot in my career. I wrote the original etcd Raft implementation (sorry!) [2] and even wrote a visualization explaining Raft[3]. It's a great tool when you need really high durability guarantees but I'm not convinced it gives you better availability guarantees than a single node that can quickly be recovered.
[1]: https://mtlynch.io/litestream/ [2]: https://github.com/goraft/raft [3]: http://thesecretlivesofdata.com/raft/
I'd say this could easily be a great engineering decision if it meets the needs of an organisation in the moment. The thing with any engineering decision is that it involves a bunch of trade-offs – doing something that you know will hit a scaling wall can be worth it, if it can get you moving faster. There will never be a perfect solution, and in these decisions I find that simplicity is often a massively under-rated benefit.
I'd even venture to say that I've seen more organisational failures as a result of optimistic over-engineering than I have optimistic under-engineering. That includes many of my own projects – ones where I've gone "oh I guess I'll do it properly this time" and then found myself six months down the line throwing out a ton of code, schema, or infrastructure I spent ages on because some requirements changed or some assumptions didn't pan out. It's hard to remember that lesson sometimes and I've been doing this for 15 years.
It's like a company I used to work for where the whole product was driven by a massive semi-structured graph database – but instead of being implemented like that it was a big ball of MongoDB JSON documents glued together with PHP by someone who was more of a domain expert than an engineer. Often pretty sketchy to work on, but an engineering decision that was a success – in the sense that it allowed that product to exist at all where it otherwise might not have.
why would an engineer want to join a company that is making these decisions? Why would the company want their users to know about these decisions?
Sign me up as both an engineer and a user. It is extraordinarily refreshing to hear "we tried a bunch of different stuff, some of it was dumb, and this is where we're at now."
Hmm, I can’t quite articulate why, but I would. Something about having the courage to be different.
If a single JSON file worked at the time, then that's clearly all they needed. It certainly hasn't seemed to limit their success.
i feel your pain dave. resume driven development and FAANG envy seem to be the dominant characteristics of far too many folks in the industry these days.
For me, it's not so much my fellow developers -- half my age, but quite bright enough to realise the benefits of simplicity -- but that the vendors of the tools our corporation has decided to use have in turn decided to implement them not just as a spaghetti-reinforced mudball, but as a spaghetti-reinforced mudball of spaghetti-reinforced mudballs.
(That these SRMB^2s are then just part of a larger spaghetti-reinforced mudball of SRMB^2s, i.e a spaghetti-reinforced mudball cubed, is of course not the vendors' fault but corporate. But then, aren't corporate systems always SRMBs? So if at least the components were simple, the end result would be only an SRM to the power of one, not three... Sigh.)