The old binaries are not necessarily simple to rebuild. Projects move / go offline. Tags get moved. Binaries for the impure systems may not be public anymore. You can't easily rebuild everything from that cache.

Depending on how bad & frequent you want outages to be and whether you want the whole thing to go down for scheduled maintenance, it can get quite a bit more complicated. A pair of servers with some level of RAID is probably the minimum to achieve any amount of availability better than a server in someone's closet. Then you need a cold backup, on top of that, no matter how many live servers you have.

I've also had much better luck with connectivity, transfer speeds, and latency, for clients in some regions, with AWS than with some other hosts (e.g. Digital Ocean). It seems peering agreement quality really matters. Not sure how the big physical server hosts fare on that front vs. AWS (which is basically the gold standard, as far as I can tell) and it's hard to find such info without trying and seeing what happens. This matters if you're trying to serve a large audience over much of the Earth. The easy single-server solution might well see some clients lost 95% of their transfer speed from your service, or not be able to connect at all, compared with AWS.

Errr... What do you mean by "easily affordable"? A quick look on Amazon says that I'd be paying something like €18.80/TB, so buying 400TB would be €7,520 - far out of the range of the majority of hobbyists. Plus a 100GBE NIC would be another €800 or so, plus you'll need all the other ancillary things like a case and a processor and RAM and cabling and whatever the monthly cost to host it somewhere, unless your ISP gives you a 100Gbps connection (which I'm sure will be extortionately expensive if they do). End result is the hardware costs alone approaching €10,000 - significantly more if you want any sort of redundancy or backups (or are you planning to run the whole thing in RAID0 YOLO mode?), plus probably low-hundreds per month, not including the time investment needed to maintain it.

S3 stores 3 separate copies across 3 AZs so for that redundancy (at 400TiB), wouldn't you need more like 6x 14x16TB servers if you're making a direct comparison?

there is human cost associated with operating this that s3 and other fully managed object stores don’t have.

there is just no reality where you build your own service for this with better reliability, less management, and less maintenance than s3 (and when i say s3 i’m including all managed object stores).

pointing at hetzner and saying, “look how cheap this is”, is missing the point.