1,783 karma · joined April 5, 2011
When we actually need to manipulate or generate something in Python, we download/upload to S3 and wrap it all in a tempfile.TemporaryDirectory() to cleanup the local disk when we're done. If you don't do this, you end up with a bunch of garbage eventually in /tmp/ you need to deal with.
We also have some longer-lived disk caches and using the data in the db and a os.stat() on the file we can easily know if the cache is up to date without hitting s3. And this cache, we can just delete stuff that's old wrt os.stat() to manage the size of it since we can always get it from s3 again if needed in the future.
Also rebasing is just so fraught with potential errors - every month or two, the devs who were rebasing would screw up some feature branch that they had work on they needed and would look to me to fix it for some reason. Such a time sink for so little benefit.
I eventually banned rebasing, force pushes, and mandated squash merges to main - and we magically stopped having any of these problems.
$ hcloud servers list hcloud: Get "https://api.hetzner.cloud/v1/servers?page=1&per_page=50&sort...": dial tcp 213.239.246.73:443: i/o timeout
as of 2025/12/22 03:26 UTC
Forkserver is probably a better default, inheriting file handles, globals, and sockets leads to a bunch of subtle bugs - I'm not sure that's even a good feature, also ymmv.
And fork() is still available, so if it breaks things, the solution would be to explicitly ask for fork() - and I'd say for most casual uses of multiprocessing, a user won't know one way or the other which is what I meant by transparent.
fork() is very hard to get right as it keeps a lot of state around from the parent process you have to cleanly discard...
The part I enjoy the most is after figuring out a solution for myself is seeing what others did on Reddit or among a small group of friends who also does it. We often have slightly different solutions or realize one of our solutions worked "by accident" ignoring some side case that didn't appear in our particular input. That's really the fun of it imho.
And most apps don't need crazy scale, they need simplicity.
You can build a boring backend on Linux VMs without containers using open-source software - it's simpler or at least a different level of complexity compared to the big clouds and orchestration systems like k8s, and honestly, it's just more fun to work on - I almost never write yaml - it's a joy.
I wrote my own deployment system using this idea - machines, roles, software and services that map to those roles, idempotent operations, and a constantly-connected async rpc system to orchestrate it all. Written from scratch in a language I like with a config language I like. My deploys are often < 10s (if I'm not waiting on webpack to build the UI) and all connect up to a chatops channel in Slack. I understand it because I wrote it all. Will it scale to infinity? Definitely not, but it's good enough for my uses.
So, I guess - just build stuff using simple primitives. Write simple software - modules and functions and a lot of stateless code. Use postgres for persistence - it's really that good. Use nginx and dns load balancing - tried and true simple architecture.
Modern Apple hardware has so much more memory bandwidth than the x86 systems they're being compared to - I'm not sure it's apples to apples.
Like if you have a large media library, you need to push maybe 10MB/s, you don't need 128GB of RAM to do that...
It's mostly just hardware porn - perhaps there are a few legit use cases for the old hardware, but they are exceedingly rare in my estimate.
The homelab group on Reddit is full of people who don't understand any of this - they have full racks in their house that could be replaced with one high-end desktop.
Commodity desktop cpus with 32 or 64GB RAM can do all of this in a low-power and quiet way without a lot more expense.
Edit, an outage of this length smells of bad systems architecture...