I consider myself reasonably fit, but I’m not sure any of the activities I do strictly require physical pain. Soreness? Fatigue? Sure. But rarely pain, unless I’ve made a mistake.
A big part of this is that people inexperienced with pushing themselves will mistake the soreness and fatigue for pain and quit. It's easy for you to say this because you claim to be "fairly fit". You have experience beyond average (at least in the US).
Also, if you are overweight, did not exercised for long, but are ambitious, that is the prime situation where it is easy to cause yourself an injury. Fairly fit means that your body is adjusted exercise at least somewhat and joints and such can withstand.