I've spent much of my career trying to optimize latency of long analytic-type queries; and I've been wrong on both sides of the issue. While it sounds stupidly tautological to say, I've come to think the important thing is being right in what I choose to optimize and when. With that bit of "wisdom" out of the way, there's a lot of ways the "premature optimization" phrase can be helpful:
* I might think I know what's important/critical-path and be wrong (it's happened)
* Users might be willing to live with slowness in some areas I didn't expect and demand performance in others
* More performant solutions might come to light later
* Particularly before product-market fit, features and their optimizations might be discarded
* New development might change the critical path
OTOH, I've found it's often beneficial to organize data and processing in a way that can take advantage of parallelism like multi-threading and vectorization. Often that's the kind of thing that's harder to do later.