I mean, more young people probably die skiing than sitting in an office.
The worst case scenario that is likely is, as said, making less money, possibly being unable to re-enter the tech field (though I consider that too unlikely to consider as well).
There's no obvious major downside to trying the ski instructor gig. I'd go for it if I thought I wanted it.