Mostly his disregard for packaging conventions and unwillingness to acomodate them. The code is usually A grade, but the odd use of the filesystem, runtime configs, daemons etc. means it will never get into any mainstream distro without heavy patching.
There's a reason why libsodium's tag line is "P(ortable|ackageable) NaCl-based crypto library".
The word of distros/packagers isn't gospel, but it counts for a lot, considering that (for better or worse) most people won't even think about using something not available as a package.