Hacker News Reader

Comment by hinkley | Hacker News Reader

Judgement or specialty.

Oddly, my first specialty was performance optimization. A predisposition met a trouble project and I spent half my time finding ways to speed up code without making it offensive. Then later projects had problems but no mandate, so I got pushback. I got really good at crypto-optimization - changes that improve performance but look like something else (cleanup, bug fixes, feature requests), and found there’s a whole quadrant of code changes that answer both legibility and speed.

There is very little magic or intuition based about any of this. It could be in a book. If anyone knows of a book that covers this, I’d love to gift it to people, because as far as I’m aware I’m seen as a weird (semiretired) street preacher on this subject. But I’m just tweaking existing recipes as it were, adding common ingredients in uncommon ways.

View on HN

I guess Michael Abrash's Graphics Programming Black Book fits the bill, though no amount of literature replaces getting your hands dirty in late 80s-early 90s level hardware+software. However, while you can do this, you're still missing the things you learned by being part of a team (of the kind which there are probably in the low hundreds right now). How much I'd give to be part of Nintendo or Id software or some other era-appropriate team figuring out how to make the early big 3D games like Mario 64 or its contemporaries on the PSX. Of course you can bust out an SDK and try it out but it probably won't be the same.

Those teams built up that kind of tacit knowledge like in this video about software texture mapping https://youtube.com/watch?v=xn76r0JxqNM

I somehow became specialist at that randomly (like you, random troubled project thrown at me at the odd chance I d find someth as a cheap new joiner turned into a lucky break and a predisposition for enjoying hunting these) and I start now to see patterns and be helpful in all troubled projects Im thrown at as a joker... to a point I lead a performance optimization mini team now...

If I were to teach, which I kinda do as a leader of the team, I d just give a few shortcut: 99% of perf issues are stupid and discoverable by profiling (which nobody does somehow, the most experienced the least likely the dev will use a profiler and the most likely he ll have 50 wrong migration ideas to make everything better with a huge budget), once you compressed all leaves of code you can find on a hot path, look at why this hot path even exist, chances are it shouldnt even be called as often, if you did that and are still too slow, redesign, multi thread more, remove business need, expand the hardware.

You start also getting an instinct on scale: a troubled team behind by half an hour on a settlement process with an exchange, so big deal, told me they wanted to move the cloud to spin up hundreds of big server with a new scalable design that would take a year to develop with 0 skill in the team yet. How many operation do they need to do in the alloted time ? 30k, with a database on the back. Well, I havent looked yet, but this sounds to me like we can try something before the mega migration lol because 30k units of work is not much and databases tend to be misused (I once fixed a million statement sent by hibernate on a SOAP call, so...) but as you see I try not to give an explanation, it s just an instinct, because prejudice is what kills perf optimization: we'll have to profile each step.

One thing I try to teach is: never migrate to a new library while calling the slow code legacy: you'll end up either failing exactly the same way or fake-suceeding on a small sample and leave the actual failure to the next team who'll call your code legacy before moving to the next fancy fad framework of the year and start the cycle of disaster over again.

One of my weirdest fixes was removing a duplicate database request on an admin page that was taking 30 seconds to load. But instead of dropping to 15 seconds, the response time dropped to 3 seconds.

I've also removed a lot of 10% calls that improved performance by 20%. Profilers don't show the cost of your code to the CPU and Virtual Memory caches. And there are new or malloc calls whose full cost is paid by another thread due to the tempo of the allocations.

As you say, looking at the count column is a huge thing. But the biggest failure I've seen is one of imagination. So many people give up when the last tall tent pole has been addressed. Even if we've only hit a 3rd of our performance goal, they believe they've tried everything and we should move on. Nobody knows how to do perf analysis with a budget. For instance, the 'current users' count should not be entitled to 10% of the page load time.

Could you write that book?

Or better yet, how about just writing a series of blog posts or a newsletter?

Don't waste time on the setup, just register something on substack.com and I'm sure a bunch of folks here will subscribe. It sounds like a fascinating subject.

Reply on news.ycombinator.com