Well it’s not like you don’t keep it in mind. Your project has some baseline performance need and you should be building with the common practices and paradigms you used to build the other parts. The point is, don’t try to guess which part might need optimization tweaks. Write it first and then prove you need to spend extra time improving something.