Just migrated from Vim to NeoVim. Downloading and configuring all the various LSPs has genuinely been one of the most frustrating aspects.
It's not a "random binary" either, it's a hosted binary for language features coming from the zed developers github release.
Even if the binary was compiled on demand when you clicked the button, were you going to go through the entire source of node to verify?
You may not vet the source of the language server, but you might want to determine which ones you are willing to trust/take the risk, and which ones you aren't.
If you want to use NeoVim, then LSP-zero + Mason was also a decent experience last I tried.
Maybe I'm missing something, but it was definitely more complicated than "just use mason".