Varnish Cache 6.0.0
varnish-cache.org
varnish-cache.org
My understanding is that fastly has rewritten the I/O model upon more industry standard event polling worker threads. They also reworked a lot of the memory model using https://github.com/fastly/uslab. None of that I/O work is upstream and it's unlikely it ever will be because of the phk@ influence and perhaps fastly sees it as IP. So be prepared to do some deep systems work if you need high scalability out of Varnish.
Apache Traffic Server is the most ready "out of the box" caching software for rolling a CDN that I've seen. If you're going to do custom development, nginx is a good I/O and HTTP model but be prepared to roll a complete cache storage model from scratch.
A better statement here would be connections are serviced by a pool of workers. Event polling is used. A single worker can and does process tens of thousands of requests a second. Workers are used to get request parallelism across multiple cores. When a request blocks, that worker will work on another request. Varnish easily beats out pure event driven software in 99perc response times because of its worker pool designed. Very puzzling that you are claiming otherwise.
> going to be dropping performance on the floor vs other I/O models as you get into the 1000s of connections
I have done benchmarks with over 100,000 active connections a second, with a mix of blocked, slow, fast connections, cache hits, misses, and passes. Performance is great, ~100us response times, very low CPU utilization.
> So be prepared to do some deep systems work if you need high scalability out of Varnish.
Not at all. Once again, I did some benchmarks not too long ago and was able to get 100Gbit/sec using a stock Varnish + stock Linux. Response time was well under 1ms per request.
Hitting high levels of scale and performance is an IO problem, not really a CPU problem. A lot of the criticism here is misinformed and focused on CPU scheduling. Not a single mention about any of the IO optimizations used in Varnish and other competing products.
Really? I would say most of us are comfortable with how Nginx and ATS are engineered and how the kernel works.
If anything, I think the concern is the fact that you are working for a publicly traded CDN yet you came to this thread armed with nothing more than a load of BS and FUD. I will say this connected a few dots in my mind and possibly added some color to what others think.
Most of the details are obviously secret, but occasionally when we found something architecturally concerning I'd talk it over with phk in IRC and he would make his own fix.
Fastly is a big complex distributed system loosely based on Varnish. There are several fantastic blog posts about Fastly's distributed architectural design, but nothing to my knowledge about anything else. Comparing single-machine scalability is completely inaccurate, mostly because you'll never know what fastly actually did or more importantly why they did it.
In general the "big co's fork of X must be superior" is a trope I'm thoroughly tired of.
Regarding thousands of connections, people on the web report Varnish handling 10k connections[1] just fine after a few tweaks, and this was ten years ago. How can it suddenly be a problem?
[1] https://varnish-cache.org/lists/pipermail/varnish-misc/2008-...
Then they describe a setup where they have to reduce the active cores from 8 to 4 because otherwise it is extremely slow.
[1] https://www.nginx.com/blog/socket-sharding-nginx-release-1-9...
> epoll/kqueue
Here is the implementation from 2.1, back maybe around 10 years ago:
https://github.com/varnishcache/varnish-cache/blob/2.1/bin/v...
https://github.com/varnishcache/varnish-cache/blob/2.1/bin/v...
> nginx thread model
https://www.nginx.com/blog/thread-pools-boost-performance-9x...
> With sendfile(2) and splice(2) you are only wiring up the activity to be done in your event loop
Why you are storing your cache on disk into individual files when its "2018 and RAM is commonly measured in many hundred GB" is beyond me. But its not even worth discussing because you will just introduce more BS, FUD, and techno babble.
> you've demonstrated to the world in this thread you have no idea how virtual memory works
Funny, I kind of consider myself an expert in virtual memory and I previously made no mention of virtual memory in any of my comments. I do consider virtual memory an extremely important part in getting high levels of performance from a cache. Did you confuse sendfile() and copying data from block devices into userspace with virtual memory?
Why am I not surprised at this reply.
With 100GB/s and very small HTTP requests you are clearly in a different group than 99.9999% of the rest and most likely also have the resources to write your own special case http cache or you could hire phk@ as a consultant.
But when it worked, it was fast.
The situation happens when an object which should be cached gets hit with a request which causes 'pass' (typically because of headers tested in builtin.vcl) and then VCL (by accident) setting the long TTL for caching on the "hit-for-pass" object.
It's an ugly corner case which we have struggled a lot with finding a solution for, which doesn't have worse side-effects than the problem we are trying to fix.
I think it is mostly a solved problem from Varnish 5.
Can anyone explain what being 'ABA-safe' means? I've not come across this before and Google is not being helpful.
My original comment was not meant to be partial to any tech and targeted toward people starting new in house CDNs, I mentioned ATS as it gives you the best ready to scale platform right now and have no incentives for sharing that knowledge. If your core business is CDN the choice opens up to from scratch or bending something else. Cambridge has shown results that should go an order of magnitude beyond current software [1][2] and a low power FPGA design that I can't find at the moment that would both be a magnitude greater in efficiency than the software mentioned in this thread.
In the mean time, using the conventional BSD stack we're closing in on 200gbps on single socket x86 hw and again hitting bus and memory bandwidth limits.
[1] https://www.cl.cam.ac.uk/research/security/ctsrd/pdfs/201408... [2] https://www.cl.cam.ac.uk/~rnw24/papers/201708-sigcomm-diskcr...
Contrary to the trollish claims below, Varnish works quite nicely, and is involved in around one fifth of all HTTP traffic globally.
WordPress is involved in even more than 20% of all HTTP traffic globally, yet it is the target of frequent criticism (whether rightfully or not is besides the point). Popularity doesn't grant free pass on valid feedback.
To clarify my own part, this was in the back of my mind http://varnish-cache.org/docs/trunk/phk/notes.html. It's written sanctimoniously and is wrong. I care a lot about the profession of systems programming and hate to see people misled. If calling bullshit is trolling I guess everyone is either a bullshitter or a troll.
I contest most the points in that page. Varnish I/O model is ignorant of TLB flushing, instruction cache, VM hints like madvise(2), and the overhead (and downright difficult job) of an OS scheduler, as well as high performance networking KPIs like load balancing SO_REUSEPORT or RSS. In a production web cache there are very good reasons you might want to park some objects in memory outside the vfs (the vfs is demand paged and also carries heavy nontrivial locking). The stuff under the header "More caches" is correct, and trivially dealt with using per worker (cpu) APIs, see counter(9) in the FreeBSD kernel for a nice example.
By the text of the page, Varnish remains 2006 software. It's now 2018 and 48 hardware threads are common on servers, RAM is commonly measured in many hundred GB, and multicore programming is beginning to mature with high quality software, libraries, books, and tooling like Intel's VTune available.
If my points didn't have some validity, Fastly would not have done what they did. That's all I have to say. I don't know nor care about phk (and find the "smear me" comment hilariously paranoid), but a lot of the technical marketing is bullshit and these problems could be fixed if he were willing to back down and revisit the bold claims.
> Varnish I/O model is ignorant of TLB flushing, instruction cache, VM hints like madvise(2), and the overhead (and downright difficult job) of an OS scheduler.
TLB flushing is a hardware problem, not a software problem. Instruction cache? Last I checked, the varnishd binary is about 2MB and uses very predictable jumping and straight function calling. Im pretty sure your going to be getting a 99% instruction cache hit rate. Madvise is used in Varnish's storage engine's, and there seems to be little interest or knowledge about how Varnish uses storage here.
> If my points didn't have some validity, Fastly would not have done what they did.
Can you expand on what Fastly did? They published a non locking memory allocator? So the claim is that memory allocation is the bottleneck and jemalloc is not suitable?
Sounds like someone might be stuck in performance hell and you are trying to draw parallels to Varnish which simply don't exist!
There is a difference between "being ignorant" and "ignoring".
I don't think there is much in modern HiPerf computing I'm ignorant of (hint: Guess who's writing one of the most intense HiPerf scientific code right now), but there sure is a lot of it I'm deliberately ignoring with respect to Varnish.
TLB flushing, instruction caching ? When your CPU is 50+% idle most of the time, that's not really what (should) worry about.
And the "downright difficult job of an OS scheduler" isn't really that hard when all threads a waiting for I/O, wake up for a few microseconds, then go back to waiting for I/O again.
And so on, and so forth...
Again: Your total lack of actual experience running Varnish shows.
You should try it some day, you might actually find it useful :-)
I've also done a lot of task-per-request (with each thread's affinity locked to a single hardware thread to avoid ripple effects), which does scale at least an order of magnitude more than threads.
I now use fiber-per-request for the best of both worlds: the easy sequential model of threads, yet the simple performance of tasks.
I can't understand why someone would defend the thread-per-request model at this point. It wasn't even a good model 10 year ago.
My personal preference is 'hitch' because I want as few lines of code involved with the certificates as possible.
For important sites, I recommend running two different SSL implementations, so that you are not dead in the water when a bad CVE in one of them comes around.