The case of watching a lot of TV is a bit different though because it's about attention. Time is limited and it makes a difference whether I do something intellectually stimulating or not. If I spend 8 hours per day creating things, writing software, learning languages, etc., that is going to develop my brain in a different way than 8 hours of watching TV does.
Now to the point about the bad look: even if the study could not prove a causal relationship, why would you even risk it? There is little to gain from watching lots of TV and you'll end up healthier and happier by prioritizing other things.