TCP Slow start plays a big role here. The fewer the packets required to carry the request, the faster the complete request can be addressed by the webserver, leading to a faster download.
http://en.wikipedia.org/wiki/Slow-start
Essentially your client waits for ACKs from the server when it sends its first packets. The greater the RTT the higher the penalty. This is part of why DC -> NJ didn't see much benefit but CA -> NJ saw a nice boost.
There was a great talk on this at Velocity Conf 2010: http://en.oreilly.com/velocity2010/public/schedule/detail/11... - I have not yet found video, but you can flip through the slides.
Checkout this visualization of the slow start and you can see how sending even one fewer green dots over a large distance could considerably change the overall time.