The card details were stored on the secure element but the pin was not and because the authentication is not done using the secure element it is trivial to bypass: http://cybersecurity.mit.edu/2012/10/google-wallet-overview-...

The current implementation of Google wallet stores your card details in a google server which protects against these attacks but is not a viable business model (google are making a loss on every transaction since they pay it and then bill the customer themselves).