Because there was fertile land on the other side.
Moving to Mars is the opposite: it is crossing the desert to go move into a frozen hellscape of radioactive desert.
> I'm sure leaving a fertile land 100 000 years ago, crossing the desert on foot
Actually, this kind of thing basically never happened, for some values of never.
Humans didn't leave Africa by crossing the Sahara desert. When we left Africa, the Sahara wasn't a desert.
Similarly, Aboriginal Australians didn't swim or canoe to the mainland, they walked there at a time when sea levels were lower and there was a land bridge to PNG.
Until very recently, nobody did crazy long crossings over uninhabitable lands or waters. Expansion happened incrementally, over habitable areas to other habitable areas.
> But eventually something pays of big time.
There is no reason to think that there is anything of value on Mars. No oil, no arable land, no liquid water, no unique unobtanium like in the movies. Just the same eighty or so elements, in roughly the same distribution. Just colder, drier, and impossibly far from everything you care about.