It's more fake because you download the app, look up your friend's number "1-555-333-2222", and your client trusts their server to actually return your friend instead of a MITM. Some asterisks there, but basically it's far from trustless.
Someone could orient an E2EE app around these trusted identities, but it'd probably not be very popular.