Doesn't a TLS terminator proxy solve this? E.g. I usually put my application services behind HTTPS-enabled nginx and it works wonderfully.
Every connection has a base cost of the TCP kernel send/recv buffer, which in our case we dropped a bit to 4kb each. So that's still 8kb per connection right there. If we terminate the SSL on a separate machine from where we handle the connection, then it means we'll be using 8kb more memory per connection. Probably even greater because nginx has its own send/recv buffers for data.
I'm sure our use-case is a unique one, most people care about raw through-put so the majority of SSL optimization has focused on lowering CPU use under high load rather than memory use under massive amounts of connections.