Personally, I've come to prefer homebrew python + pip for development on macOS (even though said python is not optimized), and clearlinux + pip for Linux and production use.
Clearlinux actually gives you a pretty simple way to install a highly optimized python and most all common libs (numpy, scipy, ML stuff, even with GPU and Intel MKL support) - then you just run a pip install -r requirements.txt to get the small, pure python stuff that you want on top that foundation.
Conda even adds advanced features, like managing multiple compiler-specific build variants in the same environment, that go way beyond anything other Python packaging tools do for this.
On top of this, you can install pip into conda environments and still use pip to manage packages if you want, or mix and match conda & pip with a pip section in conda’s environment.yml. Conda automatically manages pip installation into the activated conda environment.
Pip installing all from source is tricky to get working correctly as soon as you require a newer compiler or dependencies. Providing wheels works well but is a pain to set up correctly with good platform coverage.
That said, it doesn’t need to be one or the other. Conda integrates well with pip so you can install a base layer with conda then pip install your more esoteric dependencies.