You're forgetting that people grow out of gaming. Once you have a job and a family, gaming just isn't as fun anymore.
Much as I don't play football anymore but still watch it, I also follow some games :) and so do many friends in their late 30s / 40s with families (mostly the same ones as others in the thread, like Magic: The Gathering, League of Legends, Hearthstone, etc.).