This is why techniques like kernel bypass are used in high throughout or low latency systems like finance. Things like http://www.openonload.org/
This does tie you in to a specific network vendor but I can see the argument for moving more commodity networking hardware in this direction too.