This doesn’t seem like a real issue. Servers can handle a several megabyte executable, and CVE warnings for libraries you aren’t using can be ignored.
And, even though you might not see a way to call into the unused code, an attacker might find a way (XZ Utils).