Ask HN: What do you first look for when diagnosing someone else's slow code?
How about you?
How about you?
If the profiler points to some awful thousand-line method and the cause isn't obvious, add logging statements to record elapsed time at several points within the method; use that to narrow down the cause.
Common culprits: nested loops (sometimes hidden by looping over function calls that contain loops of their own); database abuse; calls out to third-party services.
After that it is generally a more formal investigation. Recently a coworker massively (like more than 100x) improved a vendor's product when he found, using an automated tool, that there was a huge area for improvement in the generation of links on a particular page. You might have seen syntax like
reverse('app1.view.view1')
in Djangoor
Html.ActionLink(...)
In MVC. Well this app had to do something similar, but at the scale it was now doing it, its analogous method was far too slow and was really taking up the most time on the page.Also look at data volume. There may be much more data in table(s) after long usage and long after the code was originally written.
If it is a high throughput application you should have monitoring in place that can point you to where slowness may be. Some tools will having timings on queries, loops, HTML/XML generation, etc. If not, get a profiler.
Also make sure you are up to date on the latest stable release of the stack you are running. Performance or bug fixes might help.
Bonus point if you push timing information as metrics to cloudwatch or whatever else you use. Important to look at stats like P90 for slow code that is not reproduced every time.
I escalate only if I find nothing obvious, but most issues are obvious once I have even a basic understanding of how it works.
As you said with the loop, there are only so many ways to do things. With experience, you know them and their respective failure modes.