There is no getting away from it with the new "retina" screens. You will use TTF fonts, antialiasing, and most certainly some compositing underneath it all.

But I find that rxvt is quite speedy (in both Linux and OpenBSD) not to cause any noticeable lag when using "pretty fonts". Unlike OS X, where both Terminal and iTerm are still frustratingly slow at rendering text. Sometimes I'll fire up a Linux VM just to have a fast terminal and nothing else.