We had really bizzare problems with Varnish ~6 years ago even under mild load (hundreds or requests per second). Like it would randomly let all the requests through to our origin servers, bringing them down. We struggled for months, multiple devops/sysadmins looked into it, couldn't figure it out.
But when it worked, it was fast.