Same here. Switched to iTerm for proper mouse support in Vim, and you need only click (no option key) to move your cursor in Vim if you have `mouse=a` in your vimrc.
I used to be a die-hard Terminal.app user but with Mavericks it felt incredibly sluggish doing everything from opening to listing files to editing things and this is another reason I moved to iTerm.
Be warned, though, that if you're using Vim with Solarized you and you're trapped in low color mode you want to be sure you're declared as a 256 color term and update your `vimrc` to remove the following for the colorscheme to work properly:
Remove:
* let g:solarized_termcolors = 256
* let g:solarized_termtrans = 1
* let g:solarized_contrast = "normal"
* let g:solarized_visibility = "normal"