A lot of people are comparing Google wallet and apple pay like they are the same thing - they are not. A phone with compromised firmware could leak sensitive data with Google wallet but not with Apple pay.
NFC has that "secure element" thingy, that is, a TPM, like a SIM card. Implementations may vary. I'm unaware on actual Google Wallet NFC implementation, but I heard rumors (which are not certain by any means) at least Samsung had some hardware-assisted keystore. And I doubt Visa or MasterCard would permit Android vendors to do things on a phone's base OS a manner that could be exploited with a software hack.
The current implementation of Google wallet stores your card details in a google server which protects against these attacks but is not a viable business model (google are making a loss on every transaction since they pay it and then bill the customer themselves).