Magnar had a great hack to only turn it on when you goto-line: http://whattheemacsd.com/key-bindings.el-01.html