No, not really. There are actually ways to go around that. One person who does verify a fingerprint will get a real fingerprint. Or maybe they'll just show you same numbers as to your buddy, because they control both server and client software and users have very very limited resources to control what code is shipped to them via appstores.
You see, this is all a question of trust. The only threat against which the e2ee is useful is non-trusted server admin. But then signal fans paradoxically don't trust Signal to have their unencrypted communications and simultaneously trust them to have means to get access to their messages.
So in practice a personal/corporate email / xmpp server without any e2ee gives you a better level of security than using third party service like Signal with e2ww, because in that case the attacker will have to somehow gain admin access to your server to learn anything about your communications, while in Signal's case, even if they don't compromise your encryption, they still have means to know who you are talking with, when, your IP addresses, etc.
(and don't even let me started on that crap where Signal supposedly has no ability to know who is sending you a message - it all can be logged on server side, and we're not trusting the service operator, right?)
Tldr: insisting on e2ee while fully trusting third party infrastructure is double-think at its best. If you want to be safe, use self-hosted servers. Which brings you to federation. :-D