Knuth, "Structured Programming with Goto Statements". Computing Surveys 6:4 (December 1974), pp. 261–301, §1. doi:10.1145/356635.356640
No one has said to ignore optimization, what the quote is actually telling us is to avoid falling into the trap of micro-optimizations which don't really affect the overall performance of the system. Yes, you can spend a week making the world's fastest substring-in-string function, but if you're only checking that your email address has an @ in it, that's kinda silly.
Make your programs fast by concentrating on the areas which give you the most bang for the buck and avoid falling into rat holes where you optimize something which has no impact on the system overall.