The biggest issue is that the stats are skewed by commercial aviation. When you remove that, it's relatively risky per passenger-mile.

Small planes are certainly not safer than driving. I'm a private pilot and i've studied it at length (and I love flying).

I don't have time to braindump my research, as it's slightly hard to compare fatalities per passenger mile in autos vs per flight hour with light non-commercial aviation. It's about 100,000 hours per 1.9 fatalities which includes commercial travel. It's SIGNIFICANTLY higher in small planes.

The way I summarize it to people in casual conversation is: Flying a Cessna is safer than riding a motorcycle but more dangerous than driving a car.