> There are diminishing returns but they don't really kick in until you're already pretty fit, they are only really a concern that athletes need to think about.

Unfortunately, it's people at both ends of the fitness curve that have to be careful about increased exercise frequency/intensity. On the less-fit side, the primary concern is accumulating minor injuries that reduce capacity for exercise even further leading into a downward spiral.