Honestly, the alternatives are worse. For U2F keys, you need to remember to bring your dongle with you, it has bad support on phones/mobile devices, some machines lock down ports, and there's a not-insignificant cost to buy.
You might correctly call that concern more focused on CYA of the the firm itself, but I think that's not the worse thing in the world, and the underlying crypto primitives certainly played a role here.
Meanwhile: WebAuthn (nee U2F) was literally introduced as an anti-phishing tool --- not as a database protection technology. They did U2F because people kept getting phished with code-generator authentication. The problem with TOTP is that it doesn't have a mechanism to authenticate the site asking for the code.
If you can trend towards public key based schemes there's real benefits from the IDPs prespective. Yes TOTP wasn't designed to help here, but from the IDP's perspective that just means that you should gently shift users to 2FA schemes that do provide those semantics. And U2F/FIFO is not the only other option here, just the only one that can be implemented totally in browser for the client. Duo's phone push notifications are public key crypto under the hood for one example.
The calculus is different if you're just a regular website, but the question was pointed at Google, an IDP in their own with many forms of public key schemes implemented.
The benefit of using TOTP at all far outweighs the insecurity of storing it on the same computer. And it is so low effort, I use it everywhere possible.
I have a simple script that takes a screenshot, uses zbarimg to extract the single line otpauth:// string, and appends it to my pass password database[1], which then hooks back into my browser. I somehow don't think I would bother with a hardware key for my BestBuy.com account.
Besides, my password db is itself is encrypted with GPG, and GPG can use a hardware token to be unlocked, thereby giving you more flexibility and transparency, while still benefiting from a token.
Developers can't know what configuration users have.
Point of something being a second factor is for it to be something physically distinct from the first factor, and something different in the "something you know, something you are, something you have". As long as you keep your TOTP device and password manager separate, they provide all the guarantees a second factor should.
As a bonus, they also provide one-time use tokens (both HOTP and TOTP, though HOTP is more susceptible to pre-generation of tokens to be used later), thus protecting against eavesdropping and people looking over your shoulder (or recording you).
As such, I don't buy the argument for TOTP not being a second factor auth at all.
And there are children comments here talking about how it's fine, they keep their TOTP keys in the same password database.
Yes, there are comments of people using them here that way. At my previous job, even a security IT guy did not find that an issue when I brought it up how people automated 2FA for CLI scripts. In that case, they are not even 1.5FA, they are strictly 1FA password-authentication (jumping through a few hoops to get all "components" of a password, but it's a single password protecting access).
If they are separate devices and you don't really remember your key, they are hardly something you know, IMO, so I disagree there.
In a sense, you could also extract the base cryptographic key of any hardware token (or the producer of the token might have it), and it's also "something you know", which is why I disagree. It's just trivial to extract this key for TOTP, but extraction is a separate step IMO.
If you want to do it with a protocol that enables signed messages and other stuff then the UX cannot be “type in a few numbers.”
Why not?
With TOTP, you type in a short code. The server needs to also be able to generate this code so they can verify that your code is correct. If it only has the public key then it must be able to produce the tokens to login as the user.
If you want a more complex setup where the public key can only verify that the code is valid but cannot be used to create valid codes then the codes themselves (and the protocol) become considerably more complex and aren't going to be managed by a "type six digits into a web form" UX.
I also don't think this really matters in a major sense. If somebody pwns the credential database then continuing to rely on authentication to prevent further harm is a fool's errand. "The adversary only got to extract the credential database and achieved nothing else" is not an especially common scenario today.