How could you prevent users installing their own TLS certificates, changing the ones in the binaries or patching the TLS library form inspecting the data and detecting that a ghost user joined or that the number of keys increased?

Also, I think that this is a really bad approach as anybody who manages to compromise the server can add a ghost user. This is a step backwards as E2E won't protect against active attacks anymore.