Comment by oconnor663 | Hacker News Reader

Comment by oconnor663 | Hacker News Reader

Parent Full thread

The cost of doing this in a large organization seem high though. Are there good ways to solve problems like this:

- A newbie developer adds small feature, triggering some performance test that was already close to failing. Now newbie has to learn all about the performance of this large project before they can land their first change.

- The performance test has a reasonable amount of noise. At first, 10% of test runs fail it, and people just rerun them. Eventually 90% of test runs fail, and people need to start dealing with it, but the effort required to get it all the way back down to 10% is unrealistic for any one person.

- The product is something like Microsoft Excel, that has a strong guarantee of backwards compatibility across a wide variety of hardware and OS versions that are difficult to test. Asking developers to delete code from parts of the product they're not experts in, is likely to break stuff.

> The performance test has a reasonable amount of noise. At first, 10% of test runs fail it, and people just rerun them. Eventually 90% of test runs fail, and people need to start dealing with it, but the effort required to get it all the way back down to 10% is unrealistic for any one person.

This is a tricky thing to deal with - (imo, again) how you deal with this is twofold:

1.) The automated test that measures performance needs to be rock solid - whatever you measure needs to be deterministic. Wall-clock time isn't always the way to go (although it's the easiest), maybe you profile and measure cpu cycles consumed by your application.

2.) If you capture performance over time then hopefully you can capture a trend-line that shows performance degrading from other changes that push the app past the performance bar. If your dev loop is fast enough you can isolate performance degradations to a small set of changes. Ideally if you handle #1 above this won't be much of an issue. In my experience changes that tank performance usually do so beyond the threshold of noise.

Regarding "newbie developer needs to learn all about performance" and "asking developers to delete code", that's why you need organizational buy-off. If your changes are regressing performance in a core area that is complex with a lot of legacy, maybe you shouldn't be making changes there, or the org needs to be aware of and sign off on letting your change go through. It doesn't always need to be a hard and fast "reject all changes", but there needs to be a discussion.

Reply on news.ycombinator.com