Neat! I also did a similar project but haven't invested the time in letting it run for very long. 4 million frames is about 37 game play hours, right? What were your results? Can Mario clear the first level?

When I implemented this, I scored the q-learning using real score and live count. I didn't need to add in a bonus for running right because it would randomly grab the first coin or randomly jump on the goomba quite frequently.

Also, I implemented mine using tensorflow, so it's possible I just sidestepped some bug.