If you're interested in studying CS, then it's o.k. to indulge yourself when you're in a position to do so. It is not as if it will hurt your career prospects. I'll go further and suggest that at our age, we're going to make the rest of our careers ourselves, and because you seem to have a passion for learning and teaching, that may be where you wind up.
Or not. Why not a degree in mathematics and a Master's in education?