It is so unfortunate that flying has such a credentialist mafia holding it back from more widespread use. Imagine if motorcycles had even half the regulations to ride as single seater aircraft do. Such a ridiculous state of affairs.
It ain't a perfect comparison, both have problems, but it is far easier to get a license to drive a truck hauling 20+ tons than to get a license to fly a 500 pound plane, and motorcycle licenses is basically a signature and a couple bucks away.
First, people are more likely to crash during takeoffs and landings, not in the middle of a flight. Second, it's also not just about the damage caused to others, and even if you crash in the middle of nowhere quite a few people will be busy dealing with it - in that case, remoteness is not a good thing but actually more costly. They won't just leave you and the airplane lying at the crash site until nature reclaims it all.
Furthermore, it is not just the flying itself. If that is what you wanted, the crazy early times of flight would have been for you, when everybody could just do whatever they wanted, wherever they wanted. Much of the learning for the license is the rules and procedures. Air space if very busy! I have small class D airports where there is so much traffic sometimes that I can hardly get a word in to announce myself. This will have the potential for stress and errors long after you got used to the flying itself.
I can't even begin the shortcuts someone's thinking has to take to think flying and driving should be treated the same.
It is even easier to fly a powered paraglider.
https://en.wikipedia.org/wiki/Ikarus_C42
And it flies pretty good. Not cheap though, 40 to 50K.
Someone mentionned the motorcycles. Price is the same ballpark as a Honda Goldwing Tour, a KTM RC 8C and cheaper than a 70k€ MV Agusta Superveloce serie Oro and these motorbikes only let you fly very momentarily (and with high probability that it is the last time) when you are crashing into something.