> "yup, this is the spot that I need to worry about optimizing first".
This almost never actually happens. In practice slow software of any size and complexity is sprinkled with many small inefficiencies that add up and no single thing makes a huge difference. To be performant you have to write with performance in mind from the get-go and not make many small mistakes.