I don't know about ages, but in recent non-agrarian cultures (e.g. Buffalo-hunters in North America) there were often medical specialists. But if we're getting pedantic, individualism would also have been a lot lower; people tend to look out for each other when they live together. Your “memorisation skills improve survival” argument doesn't work all that well when the memorisation required is well within the ability of almost all modern humans, if that's what their lives are centred around. (How many pieces of trivia does the average person know about celebrities? And that's not even deliberately-learnt knowledge.)
> Those without the aptitude would simply not return. Correct?
Those without the aptitude wouldn't go on such an expedition in the first place, if they had any sense in them.
You're trying hard to think of hypothetical scenarios where your hypothesis is true, and not hypothetical scenarios where your hypothesis is false. If I get to come up with loads of situations where my theory is true, I can justify any theory to myself. That's why scientists care so much about falsification.