I love this saying, though I always heard it as a more SE-specific addage: Make it work, make it right, make it fast.
Often the "fast" part is very intertwined with the "right/work" part. You can make it work/right and then realize you designed the core data structures completely wrong for performance. I know, I've made that mistake terribly before where I avoided performance until a year after a project was in flight, only to realize the fundamental API design actually made it impossible to fix. Had to basically start from scratch.
Of course it depends on if performance matters much, but it often does.
It's not about avoiding making it fast, it's about prioritizing. There's a probably apocryphal story from, I believe, The Psychology of Computer Programming (Weinberg) where a mainframe programmer opposed a new program because it wouldn't be as fast as theirs. Except their program produced garbage results, and the new one produced correct results.
The priority was in the wrong place (speed) rather than the business need (correctness).
There's absolutely nothing wrong with making your program fast, so long as it's not to the detriment of correctness or while avoiding making it correct. Fighting for performance boosts in your data access patterns or data layouts while still manipulating the data incorrectly is a fruitless endeavor.
An well-designed API should behave so that you can fudge up whatever is happening behind the scenes, so long as the outputs remain the same based on the inputs. Design for consistency and idempotency. Implementation (read: behind the scenes) details are just that, details, and subject to change. If your implementation is tied to your interface, there are bigger problems and you skipped the Make It Right part.
Most performance has to be intentionally designed in from the beginning if it matters.
So I dove in, technically I felt I could keep the API surface the same. But after about a full month of refactoring it to work with atomic CSS I found many problems. There are just some fundamental limitations to the API design you must enforce to make it work, and without such you really can't merge things properly. It's hard to explain without writing a mini-book, but needless to say the API surface very much can dictate the performance, and if you stuff your API with a bunch of features before making things fast, you may end up like me having to basically start from scratch.
My take: work/right/fast is a loop you must run many times. They also bleed into each other. Sometimes you do work/right and it feels fast, but you haven't deployed it at scale, so you never realize it's not fast. Keep your API as simple as you can, try and hit the fast part somewhat early before you add many features, and don't be afraid to take all your lessons and restart things. If fast is important to your lib, making it right must also be done in tandem with make it fast.