As of today the server could read all data, but I believe wouldn't be a huge lift to implement E2E encryption with SSH or PGP keys (or something similar).
Yeah PGP might be an option, perhaps it can be used via stdin/stdout pipes to make it an optional dependency.