The Yubikey NEO used the NXP a700x microcontroller family the last I checked (NEO-n might as well but I'm not sure).

You can read about the security features of the chip at NXP's website[1] to get a sense for what they're designed to defend against, but despite the very real possibility of a successful key retrieval attack on the card, in general it would be MUCH easier for an attacker to obtain your on-disk keychain files and figure out the password. Particularly if they don't want you to be aware the key has been compromised.

[1] http://www.nxp.com/products/identification_and_security/auth...