I've considered embedding an HSM in my concrete floor, with a USB-cable sticking out to plug into the server. :)

Assuming someone either steals your entire server or nothing, it's either that, password at boot, or some remote unlock (KSM). I doubt a burglar is going to just steal your harddrive. But I doubt they'll think of opening the floor.

Can you quantify the "harder to hack han a server"? This seems like one of those risks that is superficially true, but in practice is low enough to not matter.

Where are your E2EE keys?