I've thought about that a fair a lot recently myself, and came to a different conclusion, but like yours as a counterpoint. What I've been considering was going analog and writing down how to get to the few places I've kept my secrets and employing something like a Yubikey to access an admin account on my computer and paper instructions in an envelope in the family safety deposit box. It's not a complex as a two-person thing I read about once, but seems like a nice compromise where I can still feel like I'm protected, but the nuclear keys exist if someone ever needed them.
Where do you draw the line around data that needs to be discovered? I'm thinking about instructions to access things like bank accounts or such that they may or may not already have access to, where I'd want them encrypted but accessible. Not that I've got secret Cayman accounts or anything, but financials are usually things i want heavily encrypted, but do want family access to in case of the worst.