My worry about using my phone as the second factor is that my phone is attractive to thieves. I would personally prefer to carry around a keyring with many fobs on it.
It seems a bit cheesy, but I just put a lock screen on my phone. I figure if anyone steals it, they're unlikely to have the chops to bypass it without wiping the phone (I assume that's possible with adb at least). That combined with Google's remote wipe functionality and I'm pretty comfortable with the theft scenario.