Comment by perlgeek | Hacker News Reader

Comment by perlgeek | Hacker News Reader

And this, ladies and gentlemen, is how you not make speed comparisons.

#7 surprised me: why would 'if a:' be slower than 'if not a:'? So I ran the code myself, and ran it a few times. And lo and behold, the third digit varies greatly between runs. And the timing between the first and second variant only differ in the third digit.

I ran the first and second variant ten times, and got these values:

first: 0.138 +- 0.05

second: 0.137 +- 0.04

So there is no significant difference in my timings, though individual runs differed as much as 0.132 vs. 0.150.

Though kudos for providing the full code and the raw output. It made quite easy to reproduce!

These are the timings I measured, in octave/matlab syntax:

    t1 = [0.132781028748, 0.140866041183, 0.13587808609, 0.138177871704, 0.137129068375, 0.150414943695, 0.138676166534, 0.137069940567, 0.13484787941, 0.136043071747];
    mean(t1)
    std(t1)

    t2 = [ 0.134783983231, 0.143576860428, 0.138277769089, 0.142880916595, 0.140572071075, 0.136868000031, 0.131128787994, 0.134744167328, 0.132730007172, 0.137500047684];

    mean(t2)
    std(t2)

I was going to post this: If you do an average of `N` independent trials, your confidence interval is `O(1/sqrt(N))`.

Hence with the 10^6 trials done in the article, we must expect a couple of points deviation on the third digit.

... and since you don't know how big the constant factor is, you still have to do the experiment a few times.

Also there might be factors (that you are not aware of) that differ between several executions of the same program (maybe address space randomization got extremely lucky or unlucky in one execution? Or your OS has a bug the prioritizes processes differently when the PID is a power of 2? Who knows?), or some background processes (cron?) woke up, so even if your number of in-process repetitions is sufficiently high, you still run the thing a few times and observe if the numbers are stable.

Don't forget CPU frequency, which may slowly step up during the first few runs.

You're not supposed to run benchmarks on anything but the 'performance' CPU governor.

Though, sometimes you want to benchmark with 'ondemand' just to see how the cpu scaling affects your code.

It is known that 'ondemand' may make multi-threaded programs run slower when run on multi-core system. The problem which I have myself observed and reproduced is when two threads are alternately scheduled to run - one thread runs for some time, then wakes up the other and goes to sleep, ad infinitum. With such a workload, the scheduler/governor may enter a situation where the workload moves from one CPU core to the other (*), with no core reaching enough average load to switch it to a higher frequency.

I can't find a reference for this, but it was easy to reproduce by writing a simple program with pthreads doing what is described above. My actual application where I encountered this problem consisted of an event loop thread and a worker thread doing computational tasks (encryption). The most interesting observation was how disabling one of the two CPU cores (yes Linux can do that at runtime) sped up my application and benchmark considerably.

Taking the average is a bad idea for performance benchmarks like this running under an operating system. The minimum is much more stable. (Easy to verify experimentally: do 10x10 runs of a short benchmark and compare the variance of 10 minima to the variance of 10 averages.)

True, but the minimum might also be dependent on the various caches being primed from previous runs of the same code - a condition that will not appear during normal use.

Caches being primed is a concern, but not when choosing between minimum and average. Neither is affected much if the first run takes longer than the other nine.

IPython's %timeit magic works by running the function in three loops. It returns the minimum of the three averages. This seems like a good compromise.

I don't think it's implied that all of the listed differences are significant. I think it's designed to show you that sometimes alternatives make a difference, and other times they simply do not.

It makes the page a good resource if you're wondering if doing something a certain way has a performance impact. If it's all the same to use {} vs. dict(), why not be in the habit of using the one that's a little faster?

Moreover, the tests are organized in a meaningful way. Tests 13 and 14 are quite interesting when seen right next to each other. Showing just test 13 could be deceptive, leading you to think that operators are faster than methods. Seeing 14 right afterward, however, shows you that isn't the case, and provides some clues as to why each works the way it does. It's just enough of a hint to make you curious about what's actually going on, which I really enjoy.

I don't think the fault is with the author, but with your assumptions about the intent.

You may argue that exposition would improve this, but I think the lack of explanation is not only elegant, but better at providing insight for the reader. I don't think there's a right answer here, though.

> I don't think it's implied that all of the listed differences are significant. I think it's designed to show you that sometimes alternatives make a difference, and other times they simply do not.

The heading, "Python: faster way" doesn't suggest that to me.

And when I see comments like https://news.ycombinator.com/item?id=8814309 I feel I'm not the only one who came to conclusion that the first column was meant to be the fastest.

And how should the reader judge the significance of timing differences without knowing any variance or spreading of the measurements?

Maybe some short headings like "80% faster", "no significant difference" would make things clearer. Or color coding.

A single read from disk destroys all of them

Source: http://www.dis.uniroma1.it/~demetres/didattica/ae/upload/lec... , slide 24

Reply on news.ycombinator.com