It's really not that bad or I certainly wouldn't be using it.
The matrix home server has horrible lag spikes at some times of the day.
The e2e implementation was pretty miserable until a few weeks ago but is now pretty smooth sailing.
Been using it for 3-4 years and quite happy.