[0]: https://wiki.freebsd.org/BootTime#Past_Performance_Improveme...
> An Amazon EC2 c5.xlarge instance is being used as a reference platform and measuring the time between the instance entering "running" state and when it is possible to SSH into the instance.
The big thing is that I'm willing to sit down and earn a 50% improvement in performance by doing 10x 4% improvements in perf time. That's perseverance and foresight. Most of what I know about optimization tricks vs optimization process wouldn't quite fill up the first Gems book.
It's all about budget. If you're trying to double, quadruple the performance of something, you can't look at what the 5 slowest things are. You have to look at the thing taking 5% of the time and ask, "Does this deserve 20% of the CPU?" Because if you get 4x without touching this code, that 5% becomes 20%.
Then you don't work on the tall tent poles first, you work on the hot spots that relate to each other, and the ones you have the time and attention to do well. Because if you get a 20% improvement where a 25% improvement is possible, most bosses will not let you go back for the 2 x 2.5% later if you don't get it the first time. So then you are forever stuck with 50 x 1.5% slowdowns. Which is not a problem if you're profitable, but definitely is if you're bleeding money on hosting. Or if your main competitor is consistently 50% faster than yours.
https://lemire.me/blog/2023/04/27/hotspot-performance-engine...
You can definitely do 20-100 small optimisations and get a 2-3x speedup.
The best example I've seen of that is Nicholas Nethercote's work on speeding up the Rust compiler. A ton of tiny speed improvements that add up to 2x or 3x speed improvements overall.
I've done similar work on optimising a compiler. You're never going to get 10x without architectural changes but you can get 3x with a lot of work.
Of course the "we'll just optimise the hotspots" people have written their code so badly they need at least 10x.
By looking at a module or concern as a whole, you get to do small-to-medium architectural improvements instead of none-to-small improvements. You get a broader picture, and can pull in more small areas of improvement in the process. The fastest code is code that doesn't run, and 'tall tent-pole' optimizations miss at least half of those opportunities. In some cases they just push them around, like broccoli on a child's plate.
Many teams get to the end of this road and then have absolutely no idea what to do next. I find it infuriating for a bunch of my 'betters' to listen to someone on the business side say, "Our customers are complaining about our speed/leaving for a competitor with a faster app" and then pull up a rectangular flame chart to 'prove' "We've done all we can". You have only done the easy parts, probably not even the best parts. You might have actually broken the best parts.
These apps are 'death by a thousand cuts'. No one part of the app is particularly slow, it's just uniformly mediocre coding and an architecture that squeaks by with 'adequate' (to them, but not the business) attention to capacity planning. Either a force of nature shows up and fixes it, or the company fades away.
The only boss I ever followed to a new job had two sayings that I stole. 1) "If you want to complete a meeting in an hour you need to complete half the meeting in half an hour" and the other was about scaling:
The first order of magnitude is simple. The second is difficult. The third requires talent (sometimes tacking on "which not all teams have.")
You can interpret that a number of ways, but if your founding team isn't paying homage to the problem of scaling up to 1000 times the users you have at the beginning, you probably don't have the talent to get there (you didn't buy it, and you didn't build it along the way). I don't know what percentage of companies that claim "we did all the right things and still failed" have this problem, but I sincerely suspect it's at least 10%, which is a very big failure mode to ignore.