But think of it from a different angle, and separate each individual action. Let's define "causing physical pain" as a bad thing. Now take an example of child abuse in which physical pain is not caused - let's say fondling, or maybe consensual (ignoring subtle manipulation/etc.) sex with an older child - would you say that adding physical pain into that equation doesn't make it worse? And if adding one bad thing doesn't make it worse, what other things could you add before it does count as making it worse - murder?
Now to come back to a more specific example - if an adult has a sexual relationship with a child which the child is, at the time, untroubled by, then is there a chance that they could come out of it without any harm being done? Yes. I've no idea what the stats are, but certainly there are adults who as children had relationships with adults and who, now they are adults, still don't think there was anything wrong with it. So however likely or unlikely that is to be the case, a consensual relationship has a higher than 0% chance of doing no harm. On the other hand, a forced relationship in which the child hates it at the time, already is at 100% doing harm - even if the child gets over it and lives a happy life, there was already some harm (however serious or not serious) caused at the time. So doesn't that mean the odds suggest one is better than the other? Even if only by a tiny amount?
edit: Another thought - from a moral judgement point of view, I think it's also possible to see that there might be a difference in motivation from the abuser's point of view, who could look at a consensual relationship with a child as justifiable (i.e. deluded) vs. one who clearly knows the child doesn't want it (i.e. evil). Of course I'm sure that some of the consensual relationships (maybe the vast majority) involve adults who are just scheming and thinking about what makes their abuse easier, but some definitely justify to themselves that it's not OK. (Probably more likely with children older than 6, though.) Mental handicap example: difference between having sex with an adult who you believe to be consenting (but they're not, they just don't tell you because of their handicap) vs. having sex with an adult you know doesn't want to have sex.