It's the safety concern and it's flying over my head.
There is always a risk/benefit ratio and just because a handful of people want to spend more money so that they can fly faster from the airport to the city center than others is not worth the security implications.
You are aware how much security is involved whenever you fly?
We talk an expensive license, regular flying hours per year (also not cheap), air control, restricted air zones, very regular safety plain checks.
Every single object on a commercial plane for example is tracked, every replacement etc.
Congestion is not solved by small expensive air taxies. If a normal taxi costs x, an airtaxi has to cost 4x and more.
We the people don't need to allow everyone everything just because they have the money into do so.