If this is on a server with a few GB RAM, is there any reason not to use it as the only cache? Is there any reason not to set the bufsize to a few MB, for example (assuming memory is not a constraint)?
I have a somewhat complex setup with several micro services each served by a dynamic set of backends. Caching for some of it would be nice, but I couldn't figure out a way to use varnish (or other external cache) without ending up with 1 varnish instance per backend (a pain to maintain), introducing a single point of failure (one per service - and still a pain to maintain), or losing some of the benefit and power of haproxy as a frontend (with varnish in front of it).