One risk I've heard of second hand. If you work on your personal-interests or hobbies ... then that might cease to be fun and just become "work." I've not had that experience (I've always found the projects in my career interesting even if they have non-fun overhead at times) but it is something my wife and others have reported.