But you design for performance. The proper time to address it is at design time. That's not premature, that's the right moment.
I wish we could reserve the word "optimization" for the kinds of things you can do after implementation to improve the performance without significantly changing the design.
That is, let's continue to optimize last, but not try to make the word optimize mean address performance in general. That's not what the word means, after all.
but does it ever happen? =)
Suppose you're a snake charmer talking to an old-timer and the old-timer says, "The venomous king cobra is venomous." That's just a tautology. But if the old-timer says, "The venomous king cobra is the most venomous snake you'll ever handle and a single bite can kill an elephant," then hopefully you don't get stuck on the tautology and can see the important, actionable warning in there.
If you have a strong enough understanding of both the technology and the problem space, you really can judge the most likely bottlenecks as part of the design phase. Which can lead to radically different and better software.
You can always inform the design phase with some quick working prototypes that help validate some of the technical assumptions you are making.