I don't see what this has to do with IRC. But if all your users use your homeserver, then yes you should be fine.
I believe several are hosted on matrix.org.
However, user accounts aren’t portable yet, sadly. So the problem is more that if matrix.org disappeared then it would impact the user accounts homed on it.