Probably a combination of not using something as simple as RANSAC, and passing the output through a Kalman filter to smooth it out. (so: sufficiently advanced glue until proven otherwise)
The next thing I will do if I keep working on it will be to implement a Kalman filter to reduce shakiness. It could visually improve a lot the results since coherence of the projection between frames would make it much more attractive.
And I think this is easier to achieve with a Kalman filter rather than using a more precise estimation method.