Benchmarking Tornado vs. Twisted Web vs. Tornado on Twisted vs. Unicorn | Hacker News Reader

Benchmarking Tornado vs. Twisted Web vs. Tornado on Twisted vs. Unicorn

antoniocangiano.com

53 pointsacangiano23 comments

It would be interesting if Twisted people actually looked at the problem and tried to tune their framework to match the performance... Since someone from that team (afaik) already argued that Twisted can be just as fast, they should be inclined to prove it.

It would also be fun to see how the raw Tornado httpserver (http://github.com/facebook/tornado/blob/master/tornado/https...) performs without the Tornado web.py framework on top of it.

Testing against unicorn makes no sense whatsoever. Test against eventmachine-httpserver, that's similar in scope and featureset.

I agree that it could have been left out as it's a different beast. But I wanted to compare Tornado and the others with one of the fastest (and increasingly popular) web server from the Ruby world.

PS: I'm about to update my post with the results from Thin and evma_httpserver.

PS2: My post has been updated.

I decided to keep asynchronous, non-blocking web servers only. So I removed both Unicorn and Thin.

Be careful about reading to much into this - ApacheBench is a very limited (if convenient) benchmark that isn't very realistic of real world load.

3-4x the number of simultaneous connections (ignoring the actual performance gains) strikes me as pretty significant, even under a limited test.

The test is not just limited, it's completely meaningless because the portion of the stack he tested is not going to be your bottleneck in any real application. When will people learn that 'ab' is not a suitable benchmark for application servers...

Did not expect to see EM-httpserver kicking ass.

I think it's a c++ library (based on scanning the docs -- I could be wrong), so it's not too surprising to see it outperforming pure Python solutions.

Indeed. http://github.com/eventmachine/eventmachine/raw/b4765960db91...

  EventMachine consists of an extension library written in C++ (which can be
  accessed from languages other than Ruby)

So the post compares pure Python web frameworks with a C++ library. And not even a good library based on Boost/Asio or libevent.

Is eventmachine's c++ lib really not a "good library"? If not, why?

It reinvents the wheel. As a developer, you have to have really good reasons not to use it. Especially since Boost license is practically AS-IS, they could have taken the original source.

It is possible they are better at this problem than others, but that is very unlikely, from experience. As a developer it isn't wise to do a reliability test on a new library by yourself.

Making a portable async framework is quite error-prone because of the heterogenous situations it could face. You can witness this reading libevent's evolution over the years. And those developers are as good unix coders as you can possibly get with vast experience on the subject (since mid 90s.) I bet the Boost/Asio guys are very good C++ coders too, Boost is as close as you can get to STL.

At a glance, http://rubyeventmachine.com/ doesn't state why a new C++ async library. They only mention the Reactor Pattern, and Boost/Asio supports it:

http://www.boost.org/doc/libs/1_40_0/doc/html/boost_asio/ove...

55% of EventMachine's codebase is written in Ruby. Some parts of Tornado is written in C.

Only in the sense that Python is written in C. Tornado has a C module to expose the epoll system call on older versions of Python, but it's not needed at all with Python 2.6.

I updated my post and introduced substantial changes.

Is it possible to take advantage of this with any existing Ruby framework?

People, please stop benchmarking with ab, it does not produce accurate results. All benchmarking of this sort should be done with Siege. It's easier to use too.

Would using Siege radically change the results?

Yep, I'd expect it to halve the requests per second for every test. Paul Jones (created the Solar framework) periodically releases some pretty authoritative benchmarks for the major PHP frameworks, and here's what he has to say on the topic:

http://paul-m-jones.com/?p=421

Good to know for future reference.

His assertions are not confidence inspiring:

http://paul-m-jones.com/?p=413#comment-410123

While we're on the subject of benchmarking, has anyone run in to a problem on OS X (Leopard) where you seem to run out of ports, and the benchmark stalls at almost exactly 16,384 (2^16) requests?

The benchmark eventually resumes 10-30 seconds later. It seems OS X can't free up closed ports fast enough.

I've tried this with numerous HTTP servers (including Apache) and both ApacheBench and Siege, so it seems like it's an OS problem.

Reply on news.ycombinator.com