IMHO "code golf" would be a lot more fun and interesting (and even practical in some ways) if the goal was to produce the smallest executable and not source.
Also, I think it's not quite so impressive to see a 100-byte source file turn into a few hundred KB or few MB binary as it is to see what could be done in a 64K, 4K, or even 256 bytes binary.
However, I will say that trying to minimize the size of the source involves much of the same kind of resourcefulness and lateral thinking that minimizing executable size does. So despite the drawbacks, it really is just as fun.