You can get remarkably far using SFML or SDL, or even just a tiny wrapper over OpenGL.
Using OpenGL 2.0 with some simple "Draw yourself" objects for sprite rendering works nicely--it'll be tricky enough to teach you something, but not hardcore enough to sidetrack you from making a game.