Assuming they've already compromised the crypto bits of the chip there's nothing to gain in avoiding them since the non-crypto bits could just as well have the same compromises. Might as well just take the time/energy savings.
Tampering with the RNG probably provides the best value for an attacker, and is harder to detect.