Exactly. One of the reasons to build on top of Matrix was that it opens the path to end-to-end-encryption. Matrix E2EE (Olm) is heavily based on Signal, and wouldn't be trivial to build from the ground up.

I have a prototype running that uses Matrix-CRDT and Matrix E2EE, and it worked great. It's still a bit of a hassle to set up though (mainly, configuring the matrix sdk correctly), I hope to make that easier later this year.