previously: http://news.ycombinator.com/item?id=2629631
Also, "The client as well as the web server tested are hosted on the same computer", which is pretty poor design, to be honest.
Also, "The client as well as the web server tested are hosted on the same computer", which is pretty poor design, to be honest.
By hosting the web server and the client on the same computer, he is testing one aspect in isolation of others. This is a good thing, and is generally the way in which scientific tests advance knowledge.
Luckily this benchmark is incredibly simple. It's not a complex system as the test is using a single set of data with two pieces of software in a single contained environment; the only thing that changes is one piece of software and one configuration: the server. Separate the server/client and your test is still the same, only with extra resources for the server and client to take advantage of (and less network bandwidth and higher latency). Knowing how http clients work, and knowing how http servers work, is it possible that the client or server could be utilizing resources in such a different way after being separated as to skew the results in a significant way?
I don't believe so. Even if you saturated a 1Gbps network link, you will see differences in CPU time between processing of requests and differences in memory use, and unless they are all fast enough to saturate that link you will see some servers process more requests than others. If you want to verify this you can follow the benchmark's set-up and try on two separate machines and let us know if there's a significant difference.
Yes, maintaining constants and only altering one variable. But this test has an 2nd variable, that is altered into a special state for all of the experimentation, and then we make the leap into assuming that these results are anything other than pretty to look at when the 2nd variable is in any other position.
This is a good test for seeing which server runs best with a load testing client competing for resources on the box and using the loopback network. Thats it.
The problem is you can't test without a network stack at all, you can only test with a different network stack. Whats to say the particulars of the loopback network aren't causing more issues to be introduced to the validity of the results than the full TCP/IP stack you're trying to avoid influencing the results? Nothing, that's what.
Personally, I find this to be a welcome change from the typical "oooh-look-at-the-shiny-graphs" mentality that has become so pervasive on HN with regards to performance testing.
Keep up this spirit of questioning, and you'll discover a lot of interesting secrets. (For example, a placebo is 87% as effective as the 5 major antidepressant brands. Also, the CIE 1931 "color space" is a flawed system which is only roughly accurate.)
In general, anyone who presents statistics (of anything) should be subjected to a lot of skepticism. Scientific knowledge can only advance by truly testing one variable and only one variable.