Agreed, trust is key for a service like this. Ideally the contents need to be secure, but not so secure that you are the only one who can read them--otherwise what's the point since you're probably abducted or dead?
Like you note with your current setup, the passphrase is useless on its own.
Or store encypted file in one deadman.io switch, and the passphrase in another switch (in another account). The evil owner of said website, shouldnt be able to connect the two. (whats the chances they going to arbitarly try decoding every file they have with every passphrase they have)