Hacker News Reader

Comment by freework | Hacker News Reader

I really don't like these benchmarks. Its like benchmarking Fizzbuzz or something. Frameworks don't do anything. No one chooses a framework (at least I don't) based on performance. You choose one framework over the other because you like the API and/or language. I myself am a framework author (giotto, a python framework that was not included in these benchmarks). If my framework had been included, I'm sure it would end up dead last. When I built it, I wasn't thinking about performance, I was focusing on building a framework that would result in applications that are easy to understand/debug and fast to write.

View on HN

I agree that performance shouldn't dominate the decision, but there's no reason not to be informed by it - it can wind up mattering.

Except benchmarking is really, really hard to get right, and these benchmarks aren't really testing anything that resembles a production app.

For all non-trivial apps, by the time you get 100 req/sec your bottleneck is very likely going to be your database.

This is a good point. Especially if you only cared about how fast you can make your app. But if you want to also consider how cheap you can run your app, you need to consider how many app servers will it take to saturate the DB? 1 or 10? At certain scales for certain tasks, the hosting costs matter more than the development costs.

>But if you want to also consider how cheap you can run your app, you need to consider how many app servers will it take to saturate the DB?

Moore's law has made this sorta moot. Unless you're on Heroku, for a successful small-to-medium app, the denominator in your hosting costs is doing to be the salary of the engineer or sysadmin who tends to it.

(If you're on Heroku, then you start worrying about dynos because, with monitoring, you're paying $60 per "worker".)

This is to say, the cost in salary to properly shard a database probably outweighs a year or two of hosting for the extra two or three boxes you're spinning up; almost no one experiences explosive growth where you need to spin up dozens of new boxes overnight.

Moore's law hasn't made it moot. Running in the cloud is pretty slow and extremely expensive. Look at StackExchange for example - they used to handle a LOT of traffic on a handful of servers. Even these benchmarks say (or said) that the EC2 instance used is waaay slower than an i7 2600K.

I think you ran into your own argument: if you are very read heavy (and lots of big sites are), it's all about caching and the DB becomes irrelevant. QED really hard to get right

The general point of these benchmarks is not to resemble a full production app, but to provide a baseline measurement. From the original blog post[1]:

This exercise aims to provide a "baseline" for performance across the variety of frameworks. By baseline we mean the starting point, from which any real-world application's performance can only get worse. We aim to know the upper bound being set on an application's performance per unit of hardware by each platform and framework.

But we also want to exercise some of the frameworks' components such as its JSON serializer and data-store/database mapping. While each test boils down to a measurement of the number of requests per second that can be processed by a single server, we are exercising a sample of the components provided by modern frameworks, so we believe it's a reasonable starting point.

So, yes, these benchmarks should not be the only factor in choosing a framework, but they do provide a possibly important data point (depending on the specific scenario).

[1] http://www.techempower.com/blog/2013/03/28/frameworks-round-...

Except your assumption is complete nonsense. The more non-trivial your app is, the less the database is a bottle neck and the more the app is. The vast majority of web apps are extremely read heavy. Those apps benefit massively from caching, which completely removes the database as a bottleneck. This means you are often choosing between a language and framework combo that means paying for 50 instances vs one that means paying for 4 instances. That is a lot of money, and the fallacious notion that the slower language is more productive by virtue of being slow is silly.

If you think about it, it matters. Not that I always want the top performer, but definitely wont pick the last few.

Not sure why this is downvoted so badly, the comment is largely correct.

This benchmark is even less useful than alioth's shootout, I'm not sure why there is so much effort put into it :)

The comment is largely the authors opinion, it can be neither correct nor incorrect. It is very possible that what he feels is important and unimportant does not generalize to everyone else.

Not to mention that he assumes all the faster langs take more effort. What if you could have a lang that is a best performance and be highly productive- he assume this is not possible.

The Go code size is pretty small, in fact it might be smaller than the Rails code... I'm still trying to find all the Ruby files, Go is in one file...

I don't consider the Go size pretty small.

Mojolicious[1], Dancer[2] and Kelp[3] have set the bar for small code size for me. Not sure yet if there are smaller ones (note that there are no other files required for those apps, period)

In the same vein, Lua's OpenResty[4] looks good, as do Tornado[5], Flask[6] and Bottle[7] (although you need to tease the raw/ORM methods apart to get an idea for the last two). And of course, Sinatra[8].

There probably a lot more, especially for PHP, but I didn't feel like going through that list.

[1]: https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...

[2]: https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...

[3]: https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...

[4]: https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...

[5]: https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...

[6]: https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...

[7]: https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...

[8]: https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...

>>I don't consider the Go size pretty small.

You realize that Go implements the new template test right? Your linked ones do not (at least the ones I spot checked).

Also, Go is statically typed = win

> Also, Go is statically typed = win

Actually I find Perl's type system to make the most sense for web work:

1) Any zero length string or 0 valued int is classed as false, which is handy when checking the returns from query strings et al.

2) You can use eq for string comparison or == for numeric checking, which means which is handy has you can read values from a query string and then compare it against an int without having to do type conversion.

Don't get me wrong, I don't have anything against statically typed languages - in fact I normally prefer them. But the way Perl does type checking I find reduces the number of type problems when dealing with web development.

That all said, I much prefer working with structured types in Go than in Perl.

> You realize that Go implements the new template test right? Your linked ones do not (at least the ones I spot checked).

hello.go uses more lines defining variables and types than the entirely of many of the alternatives I posted. Obviously they will be a little longer if they implement the fortune handler, but I doubt that will really make much of a difference.

> Also, Go is statically typed = win

I'm not sure what that has to do with implementation size (which is the only thing I was addressing), but feel free to make a case.

> I don't consider the Go size pretty small. Mojolicious[1], Dancer[2] and Kelp[3] have set the bar for small code size for me.

You've basically just listed Perl 3 times though. Particularly when the guts of the code in all 3 of those examples was Perl's standard database interface (the same DBI you'd use for CGI Perl or even standalone .pl scripts).

I do love Perl for the flexibility of it's syntax and how concise the code can be. But for me the performance of Go won out. And while mod_perl* does make great gains in performance, it also makes the code a lot less portable (unlike Go). So I found myself porting my performance critical webapps over to Go

* I've not tried Mojolicious, Dancer nor Kelp so I couldn't comment on how they compare for performance.

> Yeah, the post started slightly different than it ended, and that's an artifact of that change. My next paragraph listed many more, and I tried to do multiple from each language that I looked at which had simple implementation.s

> Particularly when the guts of the code in all 3 of those examples was Perl's standard database interface (the same DBI you'd use for CGI Perl or even standalone .pl scripts).

The benchmark page clearly tags which implementations use raw SQL access and which use an ORM. These all happen to be using raw SQL. To my knowledge, none of them have a pre-bundled ORM, and I'm not sure whether the ORM tested implementations are only supposed to indicate the pre-shipped ORM.

> But for me the performance of Go won out

I wasn't trying to imply they competed on that metric, I just wanted to give some examples of much simpler implementations. What one considers small is obviously relative.

> I've not tried Mojolicious, Dancer nor Kelp so I couldn't comment on how they compare for performance.

They all look to be bottom-half of the full set of results, performance wise. Mojolicious quite a bit slower (relatively, they are all slow compared to Go) than the others, most likely because it uses it's own internal, pure Perl JSON module. There's was to fall back to the optimized C-based JSON::XS module, but I'm not sure whether that would keep with the spirit of the benchmarks.

> The benchmark page clearly tags which implementations use raw SQL access and which use an ORM. These all happen to be using raw SQL. To my knowledge, none of them have a pre-bundled ORM, and I'm not sure whether the ORM tested implementations are only supposed to indicate the pre-shipped ORM.

You miss my point. All of those examples you gave used the same core database framework and as the test was primarily a database performance test, all those 3 examples were essentially the same core Perl code.

Whether it's ORM or raw SQL is completely besides the point (though since we're on the topic, Perl's DBI basically works the same as Go's - or rather that should be the other way around given their age).

>I wasn't trying to imply they competed on that metric

Again, you missed my point. I wasn't suggestion that you were comparing the performance of the two. I was commenting on why I switched away from Perl to Go.

> I just wanted to give some examples of much simpler implementations.

Except you didn't You gave AN example (singular). It was one language; Perl.

> They all look to be bottom-half of the full set of results, performance wise.

I wouldn't trust that kind of benchmark for comparisons of Perl frameworks as setting up a Perl environment isn't as simple as compiling a Go program. With Perl, you have a number of different ways you can hook the runtimes into the web server (CGI, Apache libs, etc), pure Perl and C libraries (which you also mentioned) that significantly affect both memory usage and runtime performance and a whole boatload of config ($ENVS in mod_perl, bespoke handlers, etc) that also affect performance.

The ironic thing with Perl is despite scripts in the language being some of the most portable code on the POSIX community, running performance critical Perl webapps leads to very unportable set ups. (which was the other reason I migrated my sites to Go).

This might sound critical, but I genuinely do love Perl. I'd say it was up there as one of my favourite languages (and over the years I've learn to develop in a great number of different languages). But sadly nothing in life is perfect.

> You miss my point. All of those examples you gave used the same core database framework and as the test was primarily a database performance test, all those 3 examples were essentially the same core Perl code.

I think we are talking past each other. I listed a lot of frameworks, including three in python. I started with Perl, and added a whole bunch more. I could, and should, have presented them better.

Personally I think the fact they are using DBI is the inconsequential part. It takes up few lines of the example, and most of the other code is the specifics of the framework (although they are very similar, because they all Sinatra clones, to varying degrees). What do you expect to be different in a non-DB based test (I'm still unclear what point you are trying to make)? Their template systems are pretty simple to use as well.

> Again, you missed my point. I wasn't suggestion that you were comparing the performance of the two. I was commenting on why I switched away from Perl to Go.

That's fine, and a worthy conversation to have, I'm just trying to keep this on the topic of implementation size, since I think the performance side of the discussion is being handled well enough elsewhere.

> Except you didn't You gave AN example (singular). It was one language; Perl.

Actually I gave eight examples, three Perl, three Python, three lua and 1 Ruby. The fact there were three Perl implementations first, and listed by themselves is sort of an accident. I was really interested in how Mojolicious did, since that's my favorite at the moment, and then I checked the other Perl implementations, and then I looked for others that might be good examples. I intended for them to be taken all together, even if that's not how it seemed.

> With Perl, you have a number of different ways you can hook the runtimes into the web server (CGI, Apache libs, etc), pure Perl and C libraries ...

> he ironic thing with Perl is despite scripts in the language being some of the most portable code on the POSIX community, running performance critical Perl webapps leads to very unportable set ups.

How recent is the data this opinion is based on? My understanding is that now most (new) Perl web projects are using PSGI as a common back-end making it extremely portable, and often using pure-perl servers for performance. There's some evidence they can significantly beat mod_perl2[1].

> This might sound critical, but I genuinely do love Perl. I'd say it was up there as one of my favourite languages (and over the years I've learn to develop in a great number of different languages). But sadly nothing in life is perfect.

I was really, _really_ trying to not make it a Perl vs Go thing. It's obvious I do have a preference though. I'm glad you like Perl, it does seem to fit the mindset of certain people well, and even if they don't stick with it, they remember it fondly. :)

[1]: http://old.nabble.com/mod_perl2-vs-Starman-and-other-pure-pe...

Sorry for the brash tone of my previous posts.

I wasn't aware of PSGI nor the performance it has compared to mod_perl. That's probably one of the most interesting things I've read on here for a while (interesting in terms of it could have a direct impact on my business).

Thanks for that. :)

To find out more about PSGI/Plack then this is the best starting place (if you haven't already seen it) - http://plackperl.org

> Sorry for the brash tone of my previous posts.

I wasn't offended, just sort of confused. ;)

> Thanks for that. :)

No problem! To tell the truth I didn't really have a clue about real performance until I looked it up for that post. I use the hypnotoad (pure-Perl preforking non-blocking), server for Mojolicious for my projects, but those are mostly internal, so I didn't have to worry much about performance. I always figured I would look more into it when it mattered. I thought worst case I would deploy using PSGI on mod_perl, but I also knew from prior experience you can get pretty good performance from a pure-Perl solution.

For many systems performance matters. Claiming that there is no difference between Ruby and C is just sticking your head in the sand.

I only claimed that this is a not a useful (representative) benchmark, not that there is no difference between Ruby or C.

You can build an application in Ruby, deploy it onto 15 machines, and it will outperform the same application written in C and deployed to only one machine. Performance is more of a function of the underlying hardware than the language used to build it.

If a language is 30x less efficient than another language then you would likely need 30x more servers. Many folk are simply not prepared to spend 30x more than they need to on hardware. It's the difference between 20 servers and 600 servers.

Case in point: "How We Went from 30 Servers to 2":

http://blog.iron.io/2013/03/how-we-went-from-30-servers-to-2...

That's only true when you are using the full resources of one or more servers. If you are only using 1/100th of the server's resources, then being 30x less efficient still doesn't require any more servers.

That doesn't negate the point though, language performance matters at certain scales.

If you split a single app instance onto 15 machines, you will lose some efficiency due to network communication unless those 15 instances can work isolated without any shared data (sessions). That may not be much but worst case: you have to write that inter-machine sync code.

Reply on news.ycombinator.com