This is all well and good until you start doing anything with address sensitive replies. For example a VoIP call would need some way of anchoring the packets to a specific connection for the duration of the call (some kind of session pinning would be ideal).
I like the idea but I wonder how it performs on many use cases (like Skype or online gaming).