The main problem, as I see it, is that a sender can send to any number of recipients as many messages as they can. It's a one-to-many push model from sender to recipients. Filtering out all the trash is done on the receiver side.
This model obviously does not work. SMTP is unfixable, something new should be made.
What about making sure the sender is serious and putting the receiver in the loop of accepting the sender? This way the receiver controls what it wants to receive. If it's spam, don't receive it. Furthermore, bad senders should be penalized, so there's a form of "trust" involved as well. Also, spam-friendly ISPs and spamming botnets would become useless.
TL;DR: recipient-controlled proof-of-work puzzle given to sender.
I'm thinking of a flow, which would be something like this (just a sketch to describe the idea):
1. Sender announces intent-to-send "this is my cryptographically good identity, I want to send a message of x bytes with hash y. Please send me the proof-of-work."
2. Receiver checks if the identity is in a whitelist (trusted sender); if so, send a trivial puzzle. If not, send a default hard puzzle, 5 minutes to solve or whatever.
3. Proof-of-work sent and solved by sender, solution sent to receiver.
4. Receiver verifies proof-of-work and sends one-time token to sender.
5. Sender sends one-time token and the message.
6. Verify valid token, drop everything if invalid.
7. A human may classify the message as junk, at which point further mails from the same sender get penalized with higher puzzle strenghts.
If a 5 minutes puzzle is not enough, make it longer, the point is, the receiver controls all this.The token is used to avoid sending messages at all if the sender is scammy (to avoid potentially large messages as part of the handshake).
If the sender was legit and the message was not some unwanted marketing drivel, the recipient can whitelist the sender, letting it avoid the puzzle on later messages.
Given this, over time the situation would "stabilize" in that folks who mail each other regularily could do so with ease, and spammers would have to solve evermore complex puzzles to get their message through. It would raise the price of spamming.
Also, for new contacts, it would still work just fine: either take the default puzzle solving time penalty, or pre-emptively whitelist the sender offline (type in id).
The idea is that the cost of sending has to go skyhigh for messages which abuse the system, and valid messages should stay zero-price. Otherwise, if everyone is equal when it comes to sending messages, I'm afraid the spam problem cannot go away.
There probably are bugs in the above description, so please be gentle; my point is the SMTP model (and by extension SMTP itself) is unfixable and something new should take its place. Something more robust and more suitable to resist "byzantine faulting users" and other malicious actors.
Edit: list, typo