My rule of thumb is that as long as it doesn’t really complicate or convolute the code, write with performance in mind.
My expertise is in C# and sometimes this is as simple as using a hash set or dictionary instead of a plain list.
Other times it’s having a decent understanding of how EF (an ORM) actually works (the static typing benefits of them in C# makes ORMs very valuable when used appropriately) and the cost of round trips to any network host (including a DB).
As I think back to the people who fought me, the same group didn't like when you refactored code either because now they didn't have the code memorized. But I've found in the intervening years that making code discoverable is much more useful than trying to keep it static because coworkers are always going to need to change shit anyway to implement new features even if that feature is neither Correctness nor Performance. What you don't want is for people to hide logic from you. So if you move logic to a function call, that's fine, because it's still in the same neighborhood. But if you hoist it all the way up into a much earlier part of the call tree, you need to announce that in meetings and maybe in the documentation so people know what's what.
Once I got very confused when exactly this kind of 10k x speed up I demonstrated for a problem introduced to me as a business bottleneck then got ignored (I wasn’t on this team, though, but it’s still hard to understand the mentality and incentives there).
Then he could not get an interview anywhere. After some backchannel communication it turned out he had to remove that experience from his CV, as all employers assumed he was just lying.
I'm sure I've found factorial complexity code, at least in the design phase, but that's the worst one I've encountered in the wild.