> I don't want every single machine I use to be able to authenticate as me to anyone else.

There is the `-c` option to `ssh-add` for that reason:

     -c      Indicates that added identities should be subject to confirmation
             before being used for authentication.  Confirmation is performed
             by ssh-askpass(1).  Successful confirmation is signaled by a zero
             exit status from ssh-askpass(1), rather than text entered into
             the requester.