The problem I see is every time you drop an extra layer of innovation, you fragment the final product.
For example, Apple adds person to person payments in imessage and it suddenly just works for every single imessage user. If someone did that as a feature on top of matrix you now have a situation where some clients support it and some don't so the feature never really gets used as its too unreliable. Maybe over time the feature rolls out to more clients but there will always be some devs who think "No I don't like this feature, I won't ever add it"
I guess things are a little simpler here since it might be possible to implement new features without server changes as they may just work on top of the standard matrix spec.