Hacker News Reader

Comment by superfrank | Hacker News Reader

I haven't had a chance to dig through the code yet, but some of these results seem a bit off, especially surrounding the for loop.

For example, here are the results for the for loop 'reduce' on the small data set:

100 items - 0.030

500 items - 0.574

1000 items - 0.074

That doesn't make sense to me. How can a reduce on 1000 items be drastically less than on 500 items. Unless I'm misunderstanding something, I can only conclude that it's either A) a typo or B) they only ran this test once and the random data for the 500 test was exceptionally tough to deal with.

Either way, I would love a little more detail with the data before I trust it.

View on HN

I've seen that too, but I think that measurement is probably just garbage. Either the garbage collector worked during that period, or the whole PC had something else to do. It's just way too far off to make sense.

Haven't tested it (or even read the article :) ), but maybe some kind of JIT optimization kicks in the 1000 element case (ie afer the "loop" ran enough times) but not in the others?

The best case would then be that at the 501st element, the JIT has suddently made the code so fast each subsequent item is practically instant. In that case, the 1000 run would be as fast as the 500 run, not significantly faster.

EDIT: Maybe what happened is that the 500 case made the engine conclude that the function is called frequently enough to be optimized, and the 500 run both spent a while with the slow version and spent some time optimizing it, while the 1000 case got to exclusively use the optimized version.

Just goes to show that benchmarking complex multi-stage JITs can be hard.

True! That was just off the top of my head. I suppose you could imagine some kind of really smart dynamic analyzer which looks at the list length before starting the reduce and goes "oh, I'm going to repeat this over 500 times, better JIT it now". In that case the 1000 element case could be faster. But I don't know much about JIT compilers/analyzers and this is just theoretical, so you're likely correct :)

I've only browsed the source, but I think it's more likely that most of the results are random noise, and there's no particular reason why one case is faster than the other.

There are a ton of things one needs to do to make reliable microbenchmarks (make sure the functions are getting optimized, make sure they're not eliminated as dead code, etc.) and I don't think this repo does any of them.

yes, so I will be taking in consideration the JIT, GC and optimize function implemented in formulas, in the next push

Reply on news.ycombinator.com