The trick is: as soon as you notice it drifting, take your finger off it and let it come to a stop. If you keep pushing opposite the drift it'll never stop.

Also, if you haven't in a while, replace your cap. I replace mine every year or so and it makes it perform much better. You don't have to push as hard, which also reduces the drift problem.