The one downside I guess is that I can't (easily?) use this with things like rsync, scp & git.
scp & git are definitely in future plans. Still thinking about the best way to implement various commands which rely on ssh, so any feedback is graciously appreciated.