The statement is really a bit of a tautology: what is 'pre-mature' afterall? Well, it's any optimization that is not going to bear fruit for the user.
So the statement might as well be 'optimizations which do not improve the user experience do not improve the user experience'.
I see these sort of statements a lot. 'dont add too much salt', or 'dont drive too fast'. They dont really say much other than 'there is some max point which should not be exceeded'.
I guess my point is that it doesn't really guide us. Of course any 'pre-mature' optimization is not useful. The difficulty is knowing whether it's pre-mature, which can only gained through experience.
I think the reality is that optimizing code is not-fun. Adding features is fun. Writing tests is not-fun. If given the chance, we will find excuses to avoid the not-fun parts of software.