At least in my experience, linum makes Emacs 24 unusably slow to respond to typing. I haven't fully isolated this to "linum" vs "linum and XXX together", but when linum-mode is enabled, Emacs is noticeably sluggish.
Emacs 24 loaded with -Q (so no customizations are loaded) and linum enabled on a large file makes Emacs sluggish. Other line numbering modes also suffer; it appears to be an issue with the gutter itself.