I do often wear glasses at home, but only because it's convenient to fit my extended wear scheduled breaks (five days on, two days off, replace monthly) around being at home. If I'm already wearing contacts, I don't choose to switch to glasses at home until my schedule requires it. If I could have the same experience I have for the first five days continuously and safely, I would never take the contacts out.
But I may be an exception. I used to suffer from recurrent corneal erosion. Perhaps this causes my eyes to be less comfortable than normal when not wearing lenses.
Glasses are a big disconfort when you wear them for the first time at 18 years old, always dirty (inadvertently putting my finger on them), can't look on the sides or at the top (especially in class when taking notes), you move your neck a lot more, sun from certain angles makes you blind, etc
I agree sometimes they can become uncomfortable given certain conditions and I keep a pair of glasses around but most of the time (must be 99% for me) I find the contacts to be a better solution.