I once worked on a project that failed because the lead programmers did not want to learn how to use a database. They insisted on using an ORM incorrectly, and almost all of their code needed to be rewritten in order to handle a typical anticipated load.
Granted, the entire codebase was full of "simple" for each loops, but the reality is that if they had started by writing correct database queries, the project would have never failed.
Thus I say, optimize for your budget at the beginning of the project. You should pick design patterns that handle anticipated load. Full optimization can come in later, but if your project can't handle anticipated load at the beginning, then you are misinterpreting what this "simple, correct, fast" mantra really means.