Why? I trust that my own server is not backdoored (thus reducing the need for off-channel fingerprint verification), I have no such guarantee about Signal's server.
About your point the connection itself being MITMed, I don't exactly understand how an active attacker sitting on my connection would undermine the connection? Presumably I have a way to verify the identify of my server. Please do elaborate if I'm missing something. Regardless an interesting thought.
It's a much smaller leap to go from "I trust you to have written the code correctly" to "I also trust you to run the server well" than it is to go from "I don't trust anything" to "I trust all these billions of lines of code I didn't write".
If you self-sign a certificate instead, how do I check that the signature matches yours?
To get a (proper) secure channel, you need at least an authenticated channel first.