Kudos to the author - your projects are great.
Sorry that some of the stuff went over your head! I wanted to include longer descriptions of the tech I was using but the post was already suuuper long and I felt like I couldn't add more.
Very happy to answer any questions that you've got here!
I'm not sure how you'd simplify the architecture in a major way to be honest. There are certainly services you could use for stuff like this, but I think there you're offloading the complexity to someone else? Ultimately you need:
* A database that tracks which boxes are checked (that's Redis)
* A choice about how to put your data in your database (I chose to just store 1 million bits for simplicity)
* A way to tell your clients what the current state is (I chose to send them all 1 million bits - it's nice that this is not that much data)
* A way for clients to tell you when they check a box + update your state (that's Flask + the websocket)
* A way to tell your clients when a box is checked/unchecked (that's also Flask + websockets. I chose to send both updates about individual boxes and also updates about all 1 million boxes)
* A way to avoid rendering 1 million dom elements all the time (react-window)
Probably this is not the simplest thing to do if you want a certain degree of reliability. Should be definitely easier than writing the entire storage engine, but likely an overkill for this kind of overnight hobby projects.
you could also implement a bitset over AtomicLongArray.
more complicated: partition into x*x chunks and rw-locking those. this could be backed by an mmap'ed a million bytes for persistence, but no idea if that'd make the app disk io bound or something.
Always wonder why clearly smart people put themselves through these heroic gymnastics just to avoid learning a bit of <sys/socket.h>
Still a cool write-up B).
My takeaway from this experience is that I have no idea which of my sites is going to be popular - most of my projects are not nearly this successful - and that I should be optimizing for speed of creation, not speed of an individual project. My comparative advantage is "writing ok code and making ok decisions very quickly" and I'm going to continue to lean into that.
It may have been possible to do it all in-memory on a single large host, but then if it is unable to meet the demand or fails for whatever reason then you are completely out of luck.
I don't think it's unscalable at all - it's just that if you do this there's not a great story for adding a second machine if the first one can't handle the load (but ofc a beefy machine with a fast implementation could handle a lot of load).
When we did the go rewrite we considered just getting one beefier box and doing everything there (and maybe moving Redis state in memory), but it felt easier and safer to do a drop-in rewrite.
I mean this is basically your redis situation right? Just with a very specialized "redis".
You could scale this out, even after the pretty massive ability to scale up is exceeded. Have some front-end servers that act as a connection pooler to your datastore. Or shard the datastore, and have clients only request from the shards that they are currently looking at.
The entire timeframe of this project was 2 weeks, and the critical period (most activity / new eyes) was a couple of days.
> "is my specialized datastore gonna be faster than Redis"
Absolutely! With how efficient this code would be, you'd likely never need to scale horizontally, and in that case it is extremely easy to compete with a network hop (at least 1ms latency) versus L1 cache (<50ns)
The comparison with redis otherwise only applies once you do need to scale horizontally.
There's also the fact that redis must be designed to support any query pattern efficiently; a much harder problem then supporting just 1-2 queries efficiently.
Oh sure, yeah, I think we just agree then!
But that's based on my skillset and the timeframe of the project; I'm not disputing that the project is bounded by Redis's performance.
However... You'd need to persist those booleans somewhere eventually, of course, if you want the state to survive a process restart. And if you want multiple concurrent connections from the same box, you have to somehow allow multiple writers to the same object. And if you want multiple boxes (for redundancy, load spreading, geo distribution...), you need a way to do the writing and reading from several different boxes...
By this time you're basically building redis.