I've been using fish with ssh-agent for the last year without any problems.