An increasingly popular high-rel setup has two different SSL/TLS handlers in front of Varnish, each using a different SSL/TLS implementation.
That way a "ohh shit CVE" against either of those two implementations allow you to turn those of, and keep your site running.
If We bolted any particular TLS/SSL implementation into Varnish, you'd be down when that one got hit.
I think the threshold question will always be "Does software X make my life better?" and if it does not, you should ditch it if you can.
There is always a huge bias in reporting: People are eager to tell you why they started using your software, but they always forget the "exit-interview" when they drop it again.
The reason I hear most often for people dropping Varnish is that they have cleaned up the mess of legacy web-services, or at least transitioned it all to the New Fantastic Platform.
Other people drop Varnish for other versions of "this is now surplus to requirements" and I am totally fine with that: I dont want people to run Varnish if it doesn't make their life better.