I heard this many times and believed it, but after some time I think this is bad advice.
Here's mine:
There are two kinds of performance optimizations: free optimizations and optimizations with a cost.
The free ones are things that don't really take any extra complexity to implement, just a little thought. Using a hashed data structure instead of an array when you'll be accessing items randomly by id. Breaking early from a loop.
If you never think about these things unless you're explicitly on a profiling task, your code base will be full of "easy wins" that you have to go back and find and fix. Which will be time consuming and error prone if you do it after all your code is written.
The other group of performance fixes is ones with a complexity cost: adding a cache layer, rewriting an algorithm in a slightly more confusing way, etc.
My advice is: always pause when you are designing a new feature and think about whether there are "free" performance wins. You can implement them now with no cost, and you should. It often helps with your data design anyway, getting code that more closely matches the actual processing domain.
It doesn't have to take more than a minute. Anything bigger than that, yes: punt it til after you have profiled.