Yeah, fair enough. It's more like it could be more fun, if you're obsessed with the field, and your advisor isn't terrible, and your program doesn't encourage cutthroat competition (by e.g. admitting more students than the faculty can realistically support), and you're willing and able to live a Spartan lifestyle (in particular, no financial dependents), and ...
I'm lucky in a lot of ways and I sometimes forget that.