It’s not hard to protect an FDE key in a way that one must compromise both the TPM and the OS to recover it [0]. What is very awkward is protecting it such that a random user in the system who recovers the sealed secret (via a side channel or simply booting into a different OS and reading it) cannot ask the TPM to decrypt it. Or protecting one user’s TPM-wrapped SSH key from another user.
I have some kludgey ideas for how to do this, and maybe I’ll write them up some day.
[0] Seal a random secret to the TPM and wrap the actual key, in software, with the sealed secret. Compromising the TPM gets the wrapping key but not the wrapped key.