The Hardest Scaling Issue
blog.codeberg.org
blog.codeberg.org
> The hardest scaling issue is: scaling human power.
I just point this out because the author makes a couple head-fakes at technical issues, haha.
We transitioned from a basic stack (ext4 filesystem, Postfix, not much more) to one including so many shiny new things: Starting with the duty of maintaining own hardware (which is a journey on its own), we introduced LXC containers, Ansible, BTRFS, ZFS, Ceph and more.
In the past months, I found myself reading a lot of documentation … Still, there was no time left to dig deep into Ansible and ZFS, and by now both tools have even been mostly dropped, to reduce the stress on our team members.
For donation-powered non-profits, this uses donations that could have served better, and leads to burn-out in the team.
No matter which instance you join or if you host yourself, the data will reside on one of the few big cloud provider's systems.
We dropped Ceph (which is too complex with a minefield of issues, and few people have deep experience in it) and just use ZFS (which tons of people have experience in and performs flawlessly).
We also use Ansible since it automates and codifies our deployments. I can't imagine dropping it... in favor of what? Manually typing install commands on servers?
It is good they have lifted the covers on their service in this blog post. It screams "stay away".
My understanding is, ZFS doesn't inherently provide clustering, so each server has its own pool. If you have multiple DCs, how do you handle disaster recovery? Do you use async replication?
That can't be right -- can it? They have less than 50,000 users. For petabytes of storage to be necessary, their average per-user storage would need to be dozens of gigabytes. I've been using GitHub for more than a decade and the sum total of every repository I've published is less than that. I found a few repositories on Codeberg that are using gigabytes of storage but the majority are normal sized Git repositories.
I don't want to give GitHub free advertising by hosting my code with them nor do I want to feed their ML models. The latter is not completely avoidable since someone could mirror my repo on GitHub or they could just scrape it from Codeberg. But the former is successfully avoided by using Codeberg, even if Codeberg is hosted in the cloud.
I don't feel like the author gives enough rationale for the choices made. Making the decision so binary (Hard Way vs EvilCorp), it becomes easier to justify "Hard Way". I think if author accepts that there could be other options, maybe that will help with the problem stated.
Equinix Bare Metal Cloud[1] is an example of something in-between those two options. Not affiliated, just interested in the space.