Yes, actually you don't do install package you don't trust.
And actually even the package is code not binary, the users don't see code always. The result is user install package because they trust the author of package.
This is not good enough security position for a package manager.