Failing to recognize and live in the grey area somewhere in between (where there is a lot of fair play for diversity which sums up to an individual's personality) doesn't mark you as "introverted" like it's some disability and therefore should be respected, it's simply inconsiderate and if you're unaware of this, it's willful ignorance.
It's not against the law to be inconsiderate, nobody should force you to be different. Just don't expect others to like you, and don't get bitter when they don't.
In other words, when it comes to your "casual time" thinking you're entitled to all of it without reproach is just as bad as someone else thinking they can take any of it at a whim.