You have the greatest allowable protection when keeping your keys on hardware where you keep physical custody - e.g., on your person, or stored in physical locations where you have exclusive access (e.g., not colocated/VPS, etc).
Ultimately, you've got to decide if it bothers you to have the government - and the corporations and individuals that act on behalf of the government - archiving/indexing/searching copies of your data.
I don't believe that it's appropriate for the government to invade our privacy without due process - but this view is not universally held.
There's a core problem that's tough to solve - if people have access to strong, unbreakable crypto, then it's possible that there will be circumstances where a legitimate warrant will not produce usable information.
On the other hand, if nobody has strong, unbreakable crypto, then it's possible that there will be circumstances where privacy is invaded without a legitimate warrant.