Sounds like your network is suffering from http://www.bufferbloat.net/ .If your OS supports traffic shaping, you can probably fix this by limiting the upload speed there to a fraction under your actual bandwidth. E.g in Linux by using the tc command.