Hacker News Reader

I have to admit, I was skeptical but my complier did pick SIMD for the second and not the first and it did make a huge difference.

  % cat t.c
  #include <sys/time.h>
  #include <sys/resource.h>
  #include <stdio.h>

  double get_time()
  {
    struct timeval t;
    struct timezone tzp;
    gettimeofday(&t, &tzp);
    return t.tv_sec + t.tv_usec*1e-6;
  }

  int blah[SIZE][SIZE];
  int whatever() { static int i=0; return ++i; }

  int main()
  {
    double t0 = get_time();

    for(int i=0; i<SIZE; i++){
        for(int j=0; j<SIZE; j++){
            blah[j][i] = whatever(); // Column oriented is very slow.
        }
    }

    double t1 = get_time();

    for(int j=0; j<SIZE; j++){
        for(int i=0; i<SIZE; i++){ // Very fast: We move row-wise now. With luck, this is SIMD-vectorized by your compiler.
            blah[j][i] = whatever();
        }
    }

    double t2 = get_time();

    printf("SIZE=%5d dt1=%4.3e dt2=%4.3e\n",SIZE,t1-t0,t2-t1);

    return 0;
  }


  % for (( i=1; i<100000 ; i*=10 )) do gcc -DSIZE=$i -O t.c; ./a.out; done 
  SIZE=    1 dt1=4.179e-04 dt2=0.000e+00
  SIZE=   10 dt1=5.190e-04 dt2=0.000e+00
  SIZE=  100 dt1=5.062e-04 dt2=9.537e-07
  SIZE= 1000 dt1=4.014e-03 dt2=1.490e-04
  SIZE=10000 dt1=1.347e+00 dt2=4.349e-02

2.8G Core i7 1600MHz DDR3

n.b. there would be a difference even without vectorization

Now you could try to time it with -fno-tree-vectorize or whatever gcc uses nowadays. You might be surprised again :)

Yup yup. SIMD is a CPU-level optimization, but the problem (at size 10,000 x 10,000) is DDR4 memory-limited.

SIMD only really makes a difference at the L1 or L2 cache levels. Its a fancy micro-optimization that compilers do and can improve code speed in those cases...

But the "big" change, going from column-wise traversal into row-wise traversal, is the huge memory optimization that programmers should know about. It just so happens that SIMD-optimizations are also easier for compilers to figure out on row-wise traversal, so you get SIMD-optimization "for free" in many cases.

Can you elaborate why the second is faster in the general case? Is this an issue of row major vs column major orientation? Also does your example also hold in the absence of SIMD? Thanks.

There's at least 2 reasons (+1 non-reason) why it is slow.

1. L1 cache lines are 64-bytes long. By fetching column-wise, you are wasting the bandwidth between L1 and main-memory. L1 cache will always fetch 64-bytes. By moving "with" the cache, you allow the L1 --> Main Memory data-transfers to be far more efficient.

2. Virtual Memory is translated by the TLB before it returns the actual value. Moving within a 4kB page is more efficient than moving across pages.

The non-reason:

* Hardware prefetcher probably works, even on column-oriented data.

All of these reasons hold even if the SIMD-optimizer fails. If the SIMD-optimizer is actually working, you'll more efficiently load/store to L1 cache. But this is likely a memory-bound problem and optimizing the core isn't as important.

Make perfect sense, thanks for the concise explanations. Cheers.

If I saw that code at my company, I'd ask why someone building a rest api for web-services had to iterate through a 10000*10000 2D array while calling a method of every single cell. This isn't a use case I've seen in my 10 year career thus far.

Also, you have to understand that saving 1 or 2 milliseconds isn't important when compared to getting software built faster. For the typical use case, optimizing a for-loop like this for in-memory data isn't going to change the total performance of your program by even 0.5%. This is what so many people just don't understand. For modern web based applications, 99.9% of the time spent is doing IO, either hard-drive, data storage, or across the network. Optimizing those calls is FAR more important, than wasting time trying to save 1 millisecond in a for-loop which is iterating over 100 in-memory objects.

This is where the understanding that the practical reality of building software for a real business is very different to theoretical examples drummed up by computer scientists on discussion boards.

I see so many engineers trying to make their in-memory code faster by parallizing it, saving (if they're lucky) 1 or 2 milliseconds, but making their code far less maintainable and potentially introducing bugs, while the users wouldn't notice any practical difference in the app. Meanwhile, the app really does feel slow, because some numb-nuts made a call to some api and blocked the ui thread.

Were I work we have to process a few billion records per day. Knowing about how memory works and writing the code in a way that uses memory bandwidth efficiently, allows us to process all those records in smallish AWS instance instead of having to use Hadoop on a much more expensive cluster of instances.

Hadoop is for "big data." Big data is commonly defined as minimum of petabytes of data. That is, thousands of terabytes. Our ingest of new events to our cluster is upwards of 25B per day and we keep all that data in perpetuity.

Our hadoop cluster is relatively small at 5PB right now, but I would be impressed indeed if you were processing PB of data on a smallish AWS instance.

If you were considering using Hadoop for work that can be processed on a "smallish instance" then I'd suggest you maybe don't really understand big-data or the normal workloads for which Hadoop is intended. For example, we run ML models on our Hadoop cluster which take hours to run distributed over around 100 nodes. Let me know when this can be done on a "smallish instance."

I don't work in big data. But here's what I think of your field anyway :-)

In general, you optimize software by starting from the slowest things first, and then working your way to the faster things when you run out of things to optimize. It seems like your problems are I/O bound, and therefore the bulk of optimizations can be made by simply optimizing your I/O (either through async calls, or better understanding of what the frameworks are actually doing, etc. etc.).

And that makes sense for sure.

The thing is: the next level of optimization is not CPU-optimization, but instead memory optimization. In fact, all CPU-based optimization starts at the Main-memory level. Why?

Because memory is slower than virtually everything inside of the main CPU Core.

Which means, memory-level optimization is a far more important skill than any other CPU-based optimization technique. Modern CPU Cores operate at 4GHz easily, but RAM only responds every 100ns on server systems (that's 400 CPU-cycles of RAM Latency!)

-----------

In effect, if there's one low-level optimization any higher-level programmer should know about, it is RAM optimization. Sure, there are CPU-optimizations (SIMD registers, L1 vs L2 cache, and more), but RAM access itself is hundreds of times slower than CPU-speeds and needs to be optimized before lower-level optimizations are sought.

Once RAM access are fully optimized, then you can finally reach for the CPU-core optimizations. Instruction-level parallelism, or SIMD Registers, or what-not.

RAM is the next slowest part of your system after I/O. As such, optimizing RAM access is the most logical next step forward when your programs are running poorly.

> If I saw that code at my company, I'd ask why someone building a rest api for web-services had to iterate through a 10000*10000 2D array while calling a method of every single cell

Why not? I work with the scripting languages only and regularly hit this scenario so I won't say this is completely useless.

When you analyze a few hundred 50GB files for specific patterns, you have to go line by line in those cases shaving off milliseconds, optimizing how data should be accessed and optimized becomes valuable.

I've seen the code written by a Python web-programmer that for a given binary matrix of teams and tournaments produced top 10 rivals for each team where rivals are defined as teams that have picked the same tournaments to participate as your team.

There are tens of thousands teams and thousands of tournaments and his code was taking two days to precalculate the results for every team.

Written with minimal understanding of how slow the memory is, the new code takes less than a minute. There was no point of optimizing it further, the algorithmic changes were enough.