I hadn't thought of storing the CA key on the YubiKey. That's an interesting way to go about it.
Like the other reply though, I keep my private SSH key on the YubiKey, protected with both a PIN and a password. I can SSH from any computer as long as I can install gpg on it.