news.ycombinator.com is loading in sub-seconds, but it uses stylesheets and images hosted on ycombinator.com (a separate server) - which is taking 20s+ to respond.
Caching seems to be fine - I'm getting 304s for stylesheets and images (and they are served from local cache according to firebug and chrome dev. tools) which means it actually takes 20-30s of wait time for the server to just return the headers.
I wonder if it might be problem with Keep-Alive which is on (with 15 sec. timeout I think) - there's probably bunch of apache processes / threads sitting idle waiting for connections to close and with the amount of traffic hn gets I would imagine there's quite a few of them.