Yes, but I can get all the parties I want to talk to on my own server (if the need arises).
About your point the connection itself being MITMed, I don't exactly understand how an active attacker sitting on my connection would undermine the connection? Presumably I have a way to verify the identify of my server. Please do elaborate if I'm missing something. Regardless an interesting thought.