155 M searches / sec for one table at a time.
108 M searches / sec for two tables at a time.
88 M searches / sec for four tables at a time.
#define LOGSIZE 5
unsigned dsearch (unsigned key, unsigned * vector)
{
unsigned size = (1 << LOGSIZE) / 2;
unsigned * low1 = vector;
unsigned * low2 = vector + ((1 << LOGSIZE) / 2);
unsigned i;
for (i = LOGSIZE - 1; i != 0; i--) {
size /= 2;
unsigned mid1 = low1[size];
unsigned mid2 = low2[size];
if (mid1 <= key)
low1 += size;
if (mid2 <= key)
low2 += size;
}
if (*low2 == key)
low1 = low2;
return low1 - vector;
}
Oh well, was worth a shot..
My interpretation might be the opposite of yours: it's not that the interleaved version fails to overcome the latency, rather the apparently one-at-a-time version is already able to issue all the loads speculatively, and thus isn't being hurt (much?) by latency. If the addresses are known ahead of time, processors are great about issuing loads as soon as possible. They 'retire' them in order, but if the situation is right they can 'execute' them dozens of cycles ahead of time, even across loop iterations.
My guess would be that the slowdown from the interleaved searches is because spreading the loads across multiple cachelines actually works against you, since stalls become more rather than less frequent. Ivy Bridge can have (about?) 10 outstanding L1 requests, which it can issue two per cycle. With a 5 cycle (I think?) L1 latency, this is enough to cover up the latency. Sustaining the two loads per cycle is hard, though, since only 4 ops per cycle can retired. But there are not many other operations here, so the unrolled loop may come close.
Since we are geeking out on binary search on a Saturday night, here's an approach I developed for traversing multiple binary trees in parallel with SIMD instructions, no branches and data prefetching. It's used at Bing to calculate the document score that ranks Web search results, and traverses thousands of Machine Learning derived decision trees. It resulted in a 4x speedup over the naive approach.
But in general you are right in that once a stall happens, you aren't going anywhere.
Exactly. By running multiple interleaved branch-free binary searches at once you offer independent work to the out of order engine.
What you want is called fork & join, pretty much standard in Java 8 and Scala.