tfbvis: Visualizing TechEmpower benchmarks beyond requests per second | Hacker News Reader

tfbvis: Visualizing TechEmpower benchmarks beyond requests per second

ajdust.github.io

2 pointsemmanueloga_7 comments

Found tfbvis because I wanted to see more than RPS: specially memory footprint. The headers sortable and can be filtered.

The most memory efficient results belong to C, C++, Rust and Go, usually using around 1,1.5GB of RAM max. Java programs in the same ballpark RPS use maybe up to 4 times that much.

Not sure how much weight to put in these benchmarks though. After all, they measure performance under stress... How many web apps receive > 200,000 RPS during sustained time?

Is there a clever way to extrapolate an estimation of RAM usage under normal load?. Would be cool to see how the memory usage grows as the RPS grow, for different platforms, but I don't think the raw data has that info.

Hmm was just talking with the author of jooby and its benchmark, he pointed out he just copied one of the docker configs [1] from someone else, which explains why a lot of the benchmarks have similar memory profiles.

The jooby author also said he deployed this particular framework with as little as 512MB, which sounds reasonable. So a bunch of JVM implementations may be using "java -Xms4g -Xmx4g", so perhaps the results are not so useful, at least when it comes to the JVM.

1: https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...

TechEmpower wants to "provide representative performance measures across a wide field of web application frameworks." Do you think it succeeded in this regard? It seems like the results are pretty well received by the community in general, or at least I haven't seen a lot of people disregard these benchmarks or give them the CLBG treatment [1].

1: https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

> at least I haven't seen a lot of people disregard these benchmarks

https://old.reddit.com/r/elixir/comments/ikpgbq/benchmark_ph...

> the CLBG treatment

Please say what you mean ?

The "The Computer Language Benchmarks Game", if I remember it right, did not start with that title, and the very authors have progressively made more and more emphasis in the fact that the programs are "toy benchmarks" and not useful to compare which language is "fastest".

TechEmpower benchmarks, on the other hand, wants to "provide representative performance measures across a wide field of web application frameworks" (and proglang is a prominent part of the framework) so it seems away from the CLBG stance.

Fair enough.

> which language is "fastest"

Isn't that question too loose to be useful?

Isn't it necessary to show that performance measures are representative if we want to claim they are?

> more and more emphasis

Well, more and more literal / less and less sarcastic.

Consider —

"Some people have suggested that I summarize the results, or declare a winner. In order to satisfy this request, I have come up with a unique and subtle quantification system to score languages on their overall performance, which I call the Completely Random and Arbitrary Point System!, or CRAPS![TM], for short.

… you can link this page (with properly selected CGI parameters) from your language advocacy page as the final proof of your language's supremacy! Think of the glory."

https://web.archive.org/web/20010302143046/http://www.bagley...

Reply on news.ycombinator.com