Could strace on the ssh server help you? You would know if it's a disk issue.