I hate calling things like this a privilege. It's licensed and regulated for the safety and benefit of the people who fly and live underneath airspace (all of us).
There are too many things involving the government that have turned into privileges where it takes knowing someone or being able to afford access to obtain action.
- Having the money to afford the training for the pilots license
- Having the time to complete the training and stay current
- Having no health issues that may deny you a medical
It is pretty out of reach for most people with out the time / funds for it.
I do get what you're saying, but there are people using "privilege not a right" literally to mean that someone gets to decide whether the person is worthy beyond some standard criteria.
We’ll likely see amendments codifying those unwritten rules into existing legislation soon enough.