I'm just waiting for Matrix to get E2E going... For now, XMPP works well enough.
Also, remember that there was a time when casual users would go through the set up of local/desktop clients in order to simply read/write email - e.g. outlook express, thunderbird, apple's mail app, etc. Whether we call it setting up a server, or installing/setting up an app., eventually it won't make much difference.
This is why we've made a real effort with Olm & Megolm (http://matrix.org/git/olm/about/) in Matrix to ensure that E2E is there in the baseline, and will be turned on by default for all private rooms. (disclaimer: I work on Matrix)