No, your client asks Signal's server for the other end's public key. There's fundamentally no way for Signal's server to prove to your client that the pubkey you're encrypting for is indeed the one owned by the phone number you keyed in.
To clarify, there's no way for any software to do that. You need to somehow validate the keys' ownership out of band.
It offers a fingerprint you're supposed to validate over sneakernet but people are lazy
and Google Play offers the possibility to distribute an update only to specific e-mail addresses, so if there is a need to compromise users with a malicious update that shows another fingerprint it's really doable (or just copy the messages, like Skype was doing with TOM-Skype).