Of course you're right in many situations, but in general secrets should not be allowed in build environments. If one doesn't expend the effort to limit permissions in the production environment, however, it's no different.
It's certainly more complicated to set up than a simple "the CI system invokes a single shell script that does everything" build, but it's not actually that complicated and every CI system I've used has supported doing this.