'brew' works really well, no need for them to reinvent the wheel.
I use brew primarily only for packages that are only available on brew. I’m moving to nixpkgs for my packaging needs as soon as the M1 support drops.
Brew upgrade takes less than ten minutes or so for me. Things that influence this will be the amount and size of your installed packages and whether they have prebuilt binaries ("bottles") or if you need to build them from source. The latter is on your package maintainers.