AWS-Lc looks very promising replacement for OpenSSL these days.
It also definitely depends on your hardware, and how you use it. The main source of the performance regression is more lock contention, so it is more noticeable if you have more threads running on more cores.