Either way, you can start at any age when learning anything, especially when it's something that is primarily self-taught.
The career aspect is hard to answer. It's probably going to be harder to get an entry level job at a large publisher if you're not a fresh early 20s graduate. Then again, the industry has changed to the point where being a codemonkey at such a company isn't as coveted as it once was. It's all about small, independent companies now and you'd reasonably expect less age barriers.
Look at the engines people are using at the moment, and pick something popular. Unity is the big thing right now and definitely worth looking at from an employability standpoint going forward in the next few years as tons of small start-ups as well as established independent studios are using it. It's built on C++ and uses C#, Javascript and Boo (a Python-esque language) as scripting languages.
Also, as someone with a game development degree, my advice about going to a college is this: don't. It's a waste of time. The one exception being if you're in the US and can get in to DigiPen, as students from there seem to consistently produce interesting, high-quality stuff and do well in the industry. I have no idea if their game-centric courses are focused on programming or not though.
From my understanding (and I could be wrong), you have roughly three (3) layers: - the Game Engine that contains such things as the Rendering engine, the Net Client, the Physics Engine and the Audio Engine; - the Game Logic where is define how the object in-game interact; - the media who are the images, the sound used in the game.
Most modders are making changes only on the Game Logic and/or the media, but find an older Game Engine open-sourced and you will find example of mods that are touching directly to the Game Engine layer.
All in all, if you are interested into getting into game development, and after you have acquired some understanding of the structure of a Game as a Software, you want to start creating your game (either a recreation or something totally new). And don't worry right now about not using something that will be relevant in a year, you need to put some experience under your belt, first and foremost.
I have a young high schooler apprenticing with me right now, and he was able to get a basic 2D game going with some help in about 6 weeks, working with me for an hour a week. So it is doable, but it helps to have someone who has walked the path before.
The worse you can do is to not try. Pick a friendly language and start with a very simple game. Then work your way up...