967 karma · joined August 19, 2015
The cheaper 2.5 flash made noticeably more mistakes, for example it didn't correctly output numbers while the Pro model did.
As for OpenAI, their gpt-4o-transcribe model did worse than 2.5 flash, completely messing up names of places and/or people. Plus it doesn't label the conversation in turns, it just outputs a single continuous piece of text.
Would love to see how it performs. It supposedly has a memory bandwidth of 256 GB/sec which is about similar to a Threadripper Pro 7965WX with 8 memory channels. A Mac Studio M2 Ultra has 800 GB/sec of bandwidth though (which is RTX4080/RTX5080 territory) but is also about 2-3x more expensive at 128 GB of memory, not to mention the cost of upgrading internal storage.
I did try Erlang around 2017-2018 and quite liked it. The Joe Armstrong book is one of my favorite programming books. Perhaps I'll have a look at Elixir if I ever need to build something new from scratch.
First, these kinds of tests don't do anything useful and just show that some frameworks have more overhead than others. Once you have an app that was developed over years rather than hours and actually does a lot of stuff on every request, it's a whole different game.
Second, the kinds of applications I write rarely get more than 10-20 requests per second. If Rails would peak at 2500 rps as that YouTube video tests it at, that would be plenty. If I were writing some kind of IoT platform rather than a business app, I'd probably not start with Rails for that reason.
Third, for an actual web application, you want to consider what the user experiences, not just how fast the server responds. You can make an application feel much faster than the server response times by preloading on hover, (http) caching, async loading of things that are not visible yet, etc.
Luckily, our Rails app is very snappy and runs on hardware equivalent to an average 4 year old laptop. No issues with performance that I can't fix. No exorbitant power usage.
Also, as a niche business app, it will never have to scale as much as GitHub does.
I did learn that things wouldn't necessarily be any better if I were to use language/framework X instead of Y.
Edit: I guess I also learned that Rails does so much for you, so you can focus on building a good product. Just gotta make sure you use it as it wants to be used as much as possible.
Similarly, don't plan refactoring or architecture changes on the same board as your feature requests. Product managers will think they can change the priorities of this type of work and then they will.
The only thing that is useful to discuss in advance is scheduling when you can take the system down for maintenance.
For the rest, it's better to ask for forgiveness than to ask for permission: just do the changes that you think are technically necessary to keep everything running smoothly. Measure the outcomes and present them professionally.
Of course, with maturity as an engineer, I mean that you will not decide to do crazy, hype driven things like rewriting everything in Rust and then not doing any new features for a year. You have to be able to make trade offs and compromises to keep both sides happy.
I have not seen such a success story for any Rust web framework yet. Although I would love to see one (or more!)
I tried basically everything in this blog post with our Rails business application and none of it really works in practice. I expect the same to be true for Django.
The reason is that with Rails, you usually end up with a bunch of callbacks on models which cause "long" running transactions. For example, the application writes a record to the database, then uses the newly generated primary key to create a bunch of related records, etc. With a reasonable amount of business logic, this means that a transaction can easily run over 100 milliseconds. Not because the database is slow (it's not), but because the application may do all kinds of slow stuff in between the different statements that run in a transaction.
SQLite is single threaded when it comes to writes, so when an average write can take 100 ms, your throughput is already limited to about 10 transactions per second. IMMEDIATE transactions are indeed necessary, to wait dor the database to be ready before attwempting any transaction. Because at least it is easy to have a backoff/retry strategy before letting the framework run its transaction logic.
However, even with just a handful of active users, I needed to make sure that waiting transactions were retried a large number of times, to prevent users from getting a 500 error and requiring them to perform the same action again. However, users were complaining that the application was slow, and I had the metrics that told me the same.
Eventually I just switched to Postgres and all of the issues just disappeared. The users also immediately told me that the application became much more responsive.
I did notice in the metrics that a lot of the read operations actually became slower, as SQLite is really efficient at doing lots of small reads as compared to a client/server model database.
I do still think that SQLite is very suited for production applications, but only when it is read heavy or has very lightweight write transactions.
Thank you!