I guess I'm screwed either way. I have an office currently but I still use my headphones as much as when I didn't have one.

I don't see how that's true, but I'm open to having my mind changed. But a comfortable listening level with well-insulated headphones/IEMs for prolonged hours shouldn't be damaging. It can be tiring but having periods of quiet (home life, sleep) re-invigorates them.

Source? I've only heard of sound intensity (not duration of sound) contributing to eventual hearing loss.