If you can figure out your own ELF loader, you can figure out how to build a partially static executable that doesn't need this. You can mix static and dynamic linking. Build tooling around that is just shit.
I don't think Android is a target in most cases of this, so this would boil down to shipping 2* binaries, one glibc and one musl.
I'd need to check how musl behaves for compatibility, I'm assuming it'll be a minimum like glibc.
All of this said - you'll need to ship multiple binaries anyway, these days: x86_64 + aarch64. Possibly more…