You should be able to simple use:
nix-env -I <package_name>
Or (new flake enabled CLI tools):
nix profile nixpkgs#<package_name>
That's quite close to other package managers. You still have the massive nixpkgs repository and tools like nix-shell as an advantage.
I still have both installed as the union of their packages is better than either in isolation, but I really like having a good idea of what I have permanently installed (and why) in my nix config files, as opposed to re-spending 5 minutes in the brew docs to remember the brew incantation that will give me a reverse dependency tree and try to sort out whether I can uninstall something without breaking something else.
The other solution to this and the other is shell aliasing, make "package manager rm" uninstall regardless of the package manager, and you don't need to remember