It could also be your shell on the remote host taking it's time. Some distros have gotten a bit bloated.