In your .ssh/config file:

    Host *
    # Do not send pubkey to each and any host
    PubkeyAuthentication no
    # Use Agent
    IdentitiesOnly       yes

Then override 'PubkeyAuthentication' to yes for each known server you want to connect to.