>provide the public key

Which format?

What algorithim?

How do you prevent it from being sent to the wrong domain and enabling MITM phishing?

X.509 has a lot of issues but they've thought through a number of things. Whatever you replace it with would have similar problems. Even if you "just sent a key" it would have to be parsed in some way (Say it's RSA, you'd still have to pull the module and exponent out before you could do anything with it, if the channel is text you'd probably do some base64 decoding etc.)