So use a better terminal emulator. Rxvt-unicode allows you to pass mouse drags and scrolls directly to the terminal by holding down shift, and since it supports perl extensions, there are many ways to scroll back and yank from the terminal's buffer.
Tmux is great even if you're not using it as a process manager; it dramatically increased my productivity at the terminal (which is most of my day)