In our thought-experiment world where each address has a public key that can be used to encrypt the payload data destined for it, the public key of the source can also be used to sign the data, ensuring the sender address isn't spoofed.
If the third party amplifiers in this scenario can validate that the traffic is spoofed, it cuts out amplification attacks.