Physical locations matters a lot for uploads, mostly because client TCP stack is way less sophisticated.

For downloads high RTT can be mitigated by a congestion control that ignores constant packet-loss rates (which are common for high-rtt paths). Other tricks that you can try: fq+pacing and newer kernels with more sophisticated recovery heuristics.