* Secure handshaking requires interactivity, unless you share secrets with your actual partner (no, your CA trust store isn't enough) in advance. So your first packet would leak it.
* To return ICMP error messages ("destination unreachable"), otherwise you'd have long timeouts.
* Ratelimiting outside the server (e.g. DDOS protection). Many ISPs do actually filter source IPs. (Of course you can't on the backbone, any there are plenty shady AS.)
* Leaving it off won't help against correlation attacks.
* Most applications will need it, so it makes sense to have it in the network layer instead of coming up with incompatible implementations above.