The flipside is that I've taken a lot of time off along the way. I've taken whole years off between many jobs, I've traveled a lot, and I've spent a lot of time just doing nothing. I have some minor regrets about not making better use of my time between jobs, but I don't have regrets about taking the time off. I would have gone insane if I had worked nonstop for 20-30 years, only taking a couple weeks of vacation a year.
If I were really passionate about the work--especially if I'd launched my own business--I might not have felt burnt out or wanted time off. But I never wanted to bust my ass just for the sake of working hard, or for some nebulous future goal (although that future is now my present). If health or something else prevents me from enjoying life as much in the future, at least I've got memories of the past.