For one thing you can, in theory, optimise for the CPU you have instead of some generic target that has to run acceptably everywhere.
Here are three links that you will find informative. [0][1][2]
While I'm not using the official Gentoo binhost, I've had one running on my LAN for my own personal use for roughly a decade now. It has been possible to set up a binhost for quite a bit longer than that... that's just when I got multiple machines with the same CPU in them, so it made sense for me to set one up.
[0] <https://wiki.gentoo.org/wiki/Binary_package_guide>
[1] <https://wiki.gentoo.org/wiki/Handbook:AMD64/Installation/Bas...>
[2] <https://wiki.gentoo.org/wiki/Gentoo_Binary_Host_Quickstart>
I'm not a Gentoo user BTW but I do think building from source is attractive for the same reasons cargo (Isn't that build from source mostly?) and go do it.