Hacker News Reader

Comment by nukanse | Hacker News Reader

C++ should focus into turning SIMD into a higher level language instead of being a giant collection of intrinsics. Avx code is unreadable.

View on HN

SIMD cannot be solved at the compiler level. The capabilities of silicon SIMD implementations are fundamentally so disjoint that code using SIMD requires contextual and platform-specific data structures and algorithms.

SIMD on different silicon is incompatible at layers of abstraction far above the codegen manipulation a compiler can do.

You'll never get the full feature set of all SIMD implementations in a single generic abstraction but I'm not sure that's the only way a SIMD abstraction could provide value. There is a pretty consistent set of narrow SIMD that behaves close enough to consistently get better performance across generic targets with high level abstractions better than auto vectorized scalar code but worse than platform tuned SIMD.

The subset of SIMD that translates is so narrow that it is almost useless in real code. That’s the problem. Additionally, it is invalid for the compiler to redesign your data structures.

The ability of compilers to use SIMD is no better than it was 20 years ago. Total stagnation for all practical purposes, and the reason is that compilers fundamentally lack the context and freedom to rewrite the code architecture appropriately for SIMD optimization (even if they could).

I think a point people miss is that highly optimized SIMD code is almost never about parallelizing loops. It is about deeply probing and manipulating arbitrarily complex data structures in parallel that involve collections of unrelated types. SIMD allows you to do this, and it is fantastically efficient, but a compiler has no conceivable way of understanding this.

For this reason, SIMD optimization has often been classified as an AI-Complete problem in computer science.

Couldn't SIMD features include marking data as "the compiler is free to rearrange"?

Only in the trivial cases where the compiler already knows that. Data used inside a single function can already be rearranged. Data that is used across more than one function can be rearranged only in the case where the compiler can prove the function is only used in the one translations unit (that is files). If data is used across translations units, there is no way for the compiler to talk with the compiler of the other translations unit to agree on the data rearrangement.

There is some effort in whole program optimization which can solve the above problem in theory. However I don't know how well it works in practice.

My imagination is failing me, can you give me an example of the type of work being done where SIMD is ‘deeply probing and manipulating arbitrarily complex data structures in parallel that involve collections of unrelated types’? I don’t doubt you know what you mean, but I can’t wrap my head around it.

For me I can only think of the simplest example of SIMD being used to ‘parallelize loops’ as what SIMD is for. If it turns out I’m missing a large problem or application space for SIMD I will be excited to research it. Any pointers to topics would be appreciated.

One example is OLAP databases, which often heavily exploit SIMD for query processing. You can also turn parsing into a SIMD problem, which is applicable to things like handling XML, JSON, and CSV. Simdjson comes to mind, or parabix for XML.

The parsing stuff I am aware of, but the query processing is a big blank for me to dig into.

SIMD is often used to evaluate complex data structures in parallel. A common case is given a row with an arbitrary mix of data types and a query with an arbitrary mix of independent constraints on those data types, to evaluate all of the constraints in parallel in the same SIMD instruction(s).

The idea that SIMD is only good for parallelizing operations over arrays of the same type is empirically false and seriously understates the capabilities of good SIMD implementations. Some SQL-ish implementations work this way on row-structured data.

Most devs seriously underestimate what good SIMD is capable of.

Thanks for the response, I’ll have some reading to do in this area.

I wrote an example answer, but it was shadowbanned

Portable SIMD works well for me today, not sure why you're unable to find any use cases for it?

Is the common feature set described anywhere online? I am not in the SIMD world, and I am curious about the state of it. But I am not so committed that I am wiling to learn several different platforms’ SIMD implementations in order to suss out their commonalities.

If you want something pretty generic and easy, check out WASM's SIMD. They limit things to 128 bit widths and common operations so it can work across generic WASM targets rather than be the most efficient on x86 specifically or something.

Other targets and languages have portable SIMD targets that are a bit more advanced, but I think WASMs SIMD design is the best for generically getting your feet wet.

That's why it needs a lot of work. Writing SIMD in c++ is like writing assembly.

> SIMD cannot be solved at the compiler level.

ISPC doesn't a pretty good job of it: https://ispc.github.io/

Please stop repeating this ludicrously overstated claim.

I have personally seen several hundred use cases for SIMD that are just fine with portable intrinsics, and NOT always "platform-specific data structures and algorithms". Sure, that does happen, but not fundamentally, nor even often.

To be clear, I agree the compiler isn't solving the data layout problem for us, but portable intrinsics are indeed useful.

gemma.cpp uses them, and last I checked, has much faster long-context prefill and decode vs. llama.cpp. They hand-roll the intrinsics and presumably it is harder to land optimizations repeated across 7+ implementations/instruction sets.

In fact, C++26 made some considerable progress on that front.

https://en.cppreference.com/cpp/experimental/simd

An implementation of which exists in the following GitHub repo (albeit implemented in the xvec::simd namespace rather than std::simd...but otherwise intended to be spec-compliant).

https://github.com/intel/xvec

Speaking as a long-time C++ developer...I wasn't uninterested in contracts (which I've known about since the days of Bertrand Meyer flogging them in Eiffel), but I agree with the sentiment here that the SIMD efforts were considerably more interesting.

I wrote a comparison with Highway: https://github.com/google/highway/blob/master/g3doc/std_simd...

std::simd is unfortunately very limited.

There is std::valarray, but a google search said that only Intel's implementation uses SIMD. See [cppreference][1].

[1]: https://en.cppreference.com/cpp/numeric/valarray#Notes

There are frameworks trying to unify simd/avx, but the abstractions aren't free. At this point, we need better hardware and low level interfaces, not better abstractions.

Reply on news.ycombinator.com