Hacker News Reader

Hacker News

Comment by pimeys | Hacker News Reader

Full thread

pimeys

I run an Akkoma server for a few friends in Hetzner:

- Cloudflare DNS

- 2x CPU 2GB RAM application server with snapshots

- 2x CPU 2GB RAM postgres server with automated backups

- 2x CPU 8GB RAM elastic search server to provide a full-text search for the known Fediverse

The search is not mandatory, but super nice to have. It is about 20€ a month, not counting the domain costs due to having it already for other purposes.

Akkoma is written in elixir and is much better in handling async messaging. No need for Sidekiq or Redis. My CPU usage hovers around 6-12% and memory around 55%. Postgres uses about the same.

I've never tried Mastodon, but having been a Ruby developer in my past life, I needed to find something else to install when I joined the Fediverse. Couldn't be happier with my setup. It just works, I never really need to do any maintenance.

All the normal clients work with it.

https://akkoma.social/

View on HN

Well what kind of federation is it handling atm? Because saying Elixir is awesome means nothing until you have 1000 MAU's all tooting and hollering at the same time, and you have to federate a bunch of messages.

That's where I actually like having the sidekiq component separate and scalable. Because it means I can make use of dynamic scaling in a cloud to work through the queues, then scale down when the queues are empty.

The Mastodon web service can run on minimal resources, even with 1000 MAU's, it's only Sidekiq that needs to scale, and that also only when there is large amounts of messages to federate.

My question is, why do you need separate processes to handle mostly async operations? You need to do HTTP requests and database queries, that is mostly waiting, there is no CPU needed for this. You can do it with one process and millions of tasks operating in concurrent fashion.

I don't really know how Mastodon does this with the Sidekiq, but what I've seen people with a very small amount of activities (we're talking about hundreds of thousands, or a few millions), need to do manual scaling of Sidekiq processes or threads. As a user, you should not be tuning these metrics to handle this kind of activity. A single beam process could handle all the WhatsApp messaging back in the days, so it can easily handle our relatively tiny fediverse servers with much less users.

One of the main wins I've contributed to company after company is taking monoliths & making multiple instances of them.

Yes it can run one big process of everything, and does when I find it. But some routes or subservices are slow or hog cpu or use way more memory than the other routes. We'll have a handful of infrequently used routes that use >1GB while typically the monolith stays inder 200MB of ram. We'll have some expensive route that hog the even loop for a second. Mixing these routes in with the well behaved normal routes means we have to way over-provision, and means sporadic performance degredation once se lesser behaved routes start firing.

Separating your stuff up is a huge win. Being able to see ingest, egress, and web traffic separately & to manage them separately is a huge win. Im not sure what the win is from a single process. I have a hard time thinking of advantages. There's some small wins to memory, but a well designed forking architecture works around most of these advantages.

I'd first wish the services would be written so that they do not use too much CPU to begin with. Ruby is horrendous with CPU usage, and the concurrency/threading in Ruby runtime is nothing to brag about.

A service such as a small community Fediverse server does not need microservices and complex architecture. What I've been seeing, people are kind of struggling with Ruby and Sidekiq even when having servers that are not super big. In the past, WhatsApp was having a small group of engineers with a handful of servers, serving hundreds of millions of users with Erlang. To get people running their own Fediverse servers, it should be as easy and as cheap as possible. Akkoma is a good start, and in a few months I hope we see the first proper systems made with Rust and Go, running with the minimal amount of resources, being single process and federating everything without the user needing to do any maintenance.

I dont disagree on any point. I would still pick an implementation where I have control, management, & observability of the separate processes, such that I can see & trade off where resources are going.

Mastadon aint fast, but, this current architecture has been invaluable in letting folks degrade gracefully. Sidekiq for ingress gets backed up, but often the web service is totally responsive and egress sidekiq (if seperately configured) works fine. Picking a more effective runtime might give you a lot more headroom, but I'd still want a plan & ability to cope & degrade gracefully when the system does come under pressure, and Im not sure Erlang/Beam or any other single process system is going to have such innate ability to have the different things dealt with differently.

Bulkhead pattern is a truly great one- keep the whole ship from sinking when one part floods.

https://learn.microsoft.com/en-us/azure/architecture/pattern...

This is the gist of what makes Django and Flask and such work fine even when you’re using some GIL-affected concurrency model like gevent. Most of the waiting for your typical web server is I/O.

I need to do HTTP requests, database queries and queue processing.

Mastodon has chosen a Service oriented architecture for this, and I'm happy with that decision because it means I can keep the web and DB queries running on minimal resources, while scaling the queue processing up and down depending on current activity.

One could say that incoming http requests from end users is a fairly predictable level of activity, while queue processing is a very unpredictable level of activity. Because it's tied into people's posts, their followers, the number of instances their followers are on, and so forth.

And with Mastodon v4 the resources required for end user http requests are even fewer, because they've shifted more processing onto the client with Javascript.

So I think the freedom to be able to scale the queue processing separately is fundamental to me. I can't imagine what it would be like to run a different software where I had to scale a monolithic service.

It doesn't really matter how fast the software is because queue processing is about blocking tasks, and they're blocking because they involve contact with other services and servers on the internet.

So yes I'd love to see a sidekiq replacement in Golang, Rust, or even Elixir, but I would not like to see it bundled in with other services. Mainly because queue processing deals with blocking tasks, and incoming HTTP requests should never be blocking.

It's a Pleroma fork, so I'd say it handles it pretty well.

Can Akkoma talk with Mastodon? Otherwise I think I miss the point of your post

> Communicate with all your friends on the fediverse, no matter if they're on mastodon, pleroma, or any other fediverse software.

Probably? (Maybe it explodes if you try.)

It says it implements ActivityPub on the site so yes it should talk to Mastodon.

Talks with Mastodon, Pleroma, PeerHub, Pixelfed, Misskey, Friendica and every other software using the ActivityPub protocol.

> every other software using the ActivityPub protocol.

Even GotoSocial now that they've put special handling in to work around their bizarre key-handling design decision.

Do you know anybody running GotoSocial? I'd love to try it, but I don't see that many servers in the Fediverse yet...

Offhand, no, other than myself and that's mainly for educational purposes (the source is Go and I understand most of it[1].)

fediverse.observer points to https://gotosocial.fediverse.observer/gotosocial.sequentialr... as being open.

[1] Unlike honk, which is also in Go, but was written with little regard for readability.

What type and size of storage are you using for this ?

Ah, I forgot to add that: I use Wasabi's S3 storage for files. This is probably the most amount of money I pay and a bit wasteful. We have about one or two images per month in uploads, but Wasabi is 5 euros a month for the first terabyte of storage. Egress is free though.

Re “egress is free”.

Note that “If your monthly egress data transfer is greater than your active storage volume, then your storage use case is not a good fit for Wasabi’s free egress policy”. There are reports of bill shock from mastodon users as a result of the costs due to the policy.

https://wasabi.com/paygo-pricing-faq/#free-egress-policy

I'm all ears for a better solution. We probably will have less than a gigabyte of storage needs, and egress is what it is with some local caching from the servers. The only thing I want is to not store the files in the application server.

Looks like backblaze b2 and cloudflare as a cdn may have a more liberal egress policy. Your profile with wasabi may suit your needs, however. Another option is minio.

Yep, b2 and cf is a great combo for hobby project. Note that you still need to pay for transactions so depending on your usage pattern, b2 might range from very cheap to a nice discount.

I recommend turning on tiered cache on cf [0]. Our b2 bill went from $7 to $4.

[0] https://developers.cloudflare.com/cache/about/tiered-cache/

Cloudflare R2? https://www.cloudflare.com/products/r2/

You'd likely be in the free tier.

Maybe I'm dumb and missing something obvious - what's wrong with selfhosting and another VM if you're doing >1gb ? You already have to deal with backups on postgres VMs ?

I used to host a reverse proxy on the same box as the Mastodon instance and then have that be the origin for a bunny.net bucket. Worked great and I could make my webserver ensure that only my bucket was open.

https://www.storj.io/pricing

they have cheaper storage and up t0 150GB is free so maybe try this one as well? i am on the fence for a dirt cheap S3 myself but B2 and cloudflare R2 is in the $10-15/TB range and something cheaper would be nice for a small usecase

I use Storj. Its great.

https://www.storj.io/storj-home

Not sure about Akkoma but if architecture is similar to Mastodon, then your Wasabi bucket would include all the images and videos you see in your timelines, not just those that you upload.

At least a few years ago, Pleroma either used an actual proxy without caching or hotlinked images from other instances.

Akkoma gives you the option of caching those and serving locally or simply referring back to the original server. They have probably chosen not to cache if the number of users is small.

Reply on news.ycombinator.com