But the purpose of Anaconda is to have a easily installable set of frequently used tool for a variety of data science tasks. It's not meant to be a minimalist package.

Conda can create environments with literally nothing in them. If you’re putting the full-blown anaconda metapackage in every environment, then it sounds like you’re using conda wrong for your use case. Just tell conda to install the packages you actually need.