If the attacker got a list of passwords from a leak and your password was on it, the 2nd factor provided by the TOTP will still save you.
So, it just depends on your threat vectors. I’d rather people I support keep unique passwords alongside TOTP in a manager they’ll actually use than skip or use SMS TOTP because of a vague concern about targeted hacking of their manager.