I find it ok as long as you're using persistent connections (which also significantly reduce connection time for all other SSH sessions). Add something like this to ~/.ssh/config:
ControlMaster auto
ControlPath ~/.ssh/persist/%r@%h:%p
ControlPersist 15m