People who are really, really interested in performance will want code optimized for their CPU.
So, you would have to provide binaries for every CPU variant.
Worse, those users might also want to mix and match. For example, if they have a small SSD, they may want to optimize infrequently used tools for size, not speed, and they might want to optimize others for start up speed or security (you may want your web server to have all kinds of protections against attacks, even if that makes it slower)
=> in practice, it’s easier to let those few users who care about this build their own software.