I had this problem, although I found after I started using a tiling window manager that I gravitated back from screen (which I used obsessively) to workspaces with a fairly small (usually two, sometimes three) of xterms running. I might have a total of six running, but they're on three workspaces, and the workspaces are conceptually discreet.
Tab cycling through more than 3 terminals got to be very tedious, but it also often got to be more difficult with the multiplexer to remember the absolute ids of terminals (in cases where I was opening a bunch of them to debug a problem) than it was to visually spot the one I wanted when I had them grouped into workspaces whose ids have meaning to me.
The remote thing you mention definitely gets to be a nuisance; usually when I'm debugging a remote machine, I open an xmonad workspace and on it a couple of terminals. It feels chintzy to ssh twice, but it's also weird to go from one set of terminal selection commands (xmonad's) to another (tmux's).
Anyway this probably sounds like I'm disagreeing, I totally agree but the tiling WM's threw a wrench into my thinking about being able to absolutely select specific terminals.