I have never seen Ruby on Rails be a bottleneck, and I have been using it since version 2.
Most bottlenecks are either that database choices or poor code/design choices by developers. That is especially true today.
Most bottlenecks are either that database choices or poor code/design choices by developers. That is especially true today.
A coworker made similar claims to me about Laravel, but the framework really encourages you to do half a dozen database queries in even a pretty minimal request, and for example implemented bulk inserts as a for loop that did single inserts. If you didn't know better with an access pattern like that, you might think the database is the bottleneck long before it actually should be. Is Rails different? My sense was they are very similar.
Rails has some tooling to help with query bloat: https://guides.rubyonrails.org/active_record_querying.html#e...
That said it specifically has tools to address this that started appearing a few years ago https://github.com/rails/rails/pull/35077
The way my team handles it is to stick Kafka in between whats generating the records (for us, a bunch of web scraping workers) and and a consumer that pulls off the Kafka queue and runs an insert when its internal buffer reaches around 50k rows.
Rails is also looking to add some more direct background type work with https://github.com/basecamp/solid_queue but this is still very new - most larger Rails shops are going to be running a second system and a gem called Sidekiq that pulls jobs out of Redis.
In terms of read queries, again I think that comes down to the individual team realizing (hopefully very early in their careers) that's something that needs to be considered. Rails isn't going to stop you from doing N+1 type mistakes or hammering your DB with 30 separate queries for each page load. But it has plenty of tools and documentation on how to do that better.
The new queue you linked is database backed, but the whole point is that you want to just run a job without needing to serialize anything outside of your process. It should just schedule it onto the thread pool and give you a promise for when it's done.
The Kafka thing also seems to be an example of what I mean: in Scala I'd just make a `new Queue` with a thread safe library, and have a worker pull off and do an insert every hundred rows or so, or after e.g. 5 ms have passed, whichever is first. No extra infrastructure needed, minimal RAM used, your queueing delay is in the single digit ms, and you get the scaling benefits. Takes maybe 10-20 lines of code.
You can then take that and abstract it into a repository pattern so that you could have an ORM that does batching for you with single item interfaces (for non-transactional workflows), but none of them seem to do this.
And again maybe I'm just not understanding but I really like having our background processes handled completely separately from our main web application. Maybe its just the peace of mind knowing that I can scale them independently of each other.
And, like, Mastodon apparently uses a queue to do things like send new user registration emails. Why not just send the email from the new user request handler? Then if there's an error, you can tell the user in the response instead of saying "okay you should get an email" and then having it go into the ether. I was under the impression this had something to do with not wanting to tie up the HTTP worker because you want it to quickly get back to doing HTTP requests, but if it can concurrently process requests, there's no issue.
Similarly they have an ingest queue for other federated servers sending them updates. But if things are fast, why wouldn't they just process the updates in the HTTP handler? You don't need a reliable queue because if e.g. you crash, the other side will not get their HTTP response, and they'll know to retry.
That said, in my experience, CPU is often the ultimate bottleneck with PHP, Ruby, Python, and.. Like everything. Over the years serializers have often been a pain point; XML in PHP and RoR, and the Rails "serializers" currently. Any sort of mapping or hydration(which is a LOT of what happens in web apps) is comparatively slow, often order of magnitude or more, over something like nodejs, C#, golang, and etc.
> Most bottlenecks are either that database choices or poor code/design choices by developers
Perhaps in sheer quantity, but with experience those are often low hanging fruit. After those are addressed you are left with the pain of the language and framework inefficiencies.