Hacker News Reader

People, please stop benchmarking with ab, it does not produce accurate results. All benchmarking of this sort should be done with Siege. It's easier to use too.

View on HN

While we're on the subject of benchmarking, has anyone run in to a problem on OS X (Leopard) where you seem to run out of ports, and the benchmark stalls at almost exactly 16,384 (2^16) requests?

The benchmark eventually resumes 10-30 seconds later. It seems OS X can't free up closed ports fast enough.

I've tried this with numerous HTTP servers (including Apache) and both ApacheBench and Siege, so it seems like it's an OS problem.

Would using Siege radically change the results?

Yep, I'd expect it to halve the requests per second for every test. Paul Jones (created the Solar framework) periodically releases some pretty authoritative benchmarks for the major PHP frameworks, and here's what he has to say on the topic:

http://paul-m-jones.com/?p=421

His assertions are not confidence inspiring:

http://paul-m-jones.com/?p=413#comment-410123

Good to know for future reference.

Reply on news.ycombinator.com