Instead of using Yubikey through GPG to replace the SSH key one can also use the PKCS#11 module they provide: https://developers.yubico.com/PIV/Guides/SSH_user_certificat... This requires replacing GNOME keyring with SSH agents, though.

One can also use one of the 20 additional slots (82-95) the Yubikey 4 provides over the Yubikey NEO by replacing the '9c' with '82'.