I use different passwords for almost every site and so if one is compromised my risk is fairly limited. If the password manager was compromised in some way it would raise my exposure significantly.
Not all, the rarely used (but needed) ones are stored, and I am old school, they are printed and stored in a safe. But the ones I use every week, yea, I remember them. Part of my ability to do that is I also shutdown accounts if I am not actively using them to keep it a manageable number. Mostly because there is no way I could remember too many at once and not screw up by having some easily guessable pattern.