When 98% of your users only have the binary and not the source code it means 98% of your users won't even be aware that they need a new version that isn't vulnerable.
Remember the vulnerability disclosure will be something like "Log4Go has a vulnerability!!", not that the binary apps you've downloaded have a vulnerability. So if it affects 200 projects, every one of those projects will also have to publish vulnerability notifications independently.
And frankly 80% of those projects will likely be abandoned or at least neglected after a couple of years, leaving many people with vulnerable binaries they don't even know to check for vulnerabilities.
In other words, they very advantage of Go (easy to distribute binaries directly!) is a security disadvantage. If you instead installed the app using a package manager, the package manager could at least notify you that you have a package with a security issue. But that's nothing new; security and ease-of-use are often at odds.