janwas

I like taking the trimmed mean of 10-20 runs, or if a run is quick, the (half-sample) mode of more runs. See robust_statistics.h.

Right, though the out of bounds depends on VL, which makes it less generally useful than a known 128-bit limit. It also requires at least one extra instruction to get a mask.

Good to know. There is more to shuffle instructions than just their speed, though. Arm TBX leaves out-of-bounds elements unchanged, and x86 PSHUFB can also zero the result and implicitly clear bits 4..6. Some SW does make use of that, so it seems unfortunate that RVV doesn't get either of those without extra effort.

Great writeup! One comment: PSHUFB does seem important, rather than just "nice to have".

> This should be mostly mitigated because the base V extension only has the full lane-crossing-gather, hence a lot of software will use lane-crossing operations and force vendors to implement it competitively.

Is that actually what's happening? From the discussion you link (https://github.com/llvm/llvm-project/pull/104574), it seems like the X280 behavior is currently still widespread. Do we know when the competitive perf might materialize?

This advice seems specific to an interpreter with "hundreds of slow paths" :)

Sure, we generally don't want branches in the code.

What makes intrinsics really interesting is that you can use wrapper libraries such as our Highway, then the code actually is portable and efficient.

Or perf/TCO, or availability? :)

Good points. gemma.cpp can also be an interesting benchmark, it also uses Highway.

FWIW I ran a quick test of gemma.cpp on M3 Pro with 8 threads. Similar PaliGemma inference speed to an older AMD (Rome or Milan) with 8 threads. But the AMD has more cores than that, and more headroom :)

Most of the SIMD code I write can consume tens of GB/s BW per core. From this perspective, it seems that 192 cores make for an imbalanced system.

That's indeed short for some actual uses such as summarization, but AFAIK many/most? evals involve generating less than 200.

We (gemma.cpp) recently started accumulating softmax terms into f64. There is at least one known case of this causing differing output, but after 200 tokens, hence unlikely to be detected in many benchmarks.

Does anyone have experience with higher-precision matmul and whether it is worthwhile?

Thanks! Yes, that (rminimax/ratapprox) looks very interesting. I got the impression that this was separate from Sollya.

Fair point about the correct rounding. We'd be fine with several ULPs.

Possible misunderstanding: I mean a rational function in the sense of Padé approximation or CF [1], not just representing individual numbers as p/q. I did not find anything related to this in Sollya [2].

[1]: https://www.jstor.org/stable/2157229 [2]: https://www.sollya.org/releases/sollya-8.0/sollya-8.0.pdf

hm, why not Chebfun? Result is a rational polynomial so we have to divide, but that seems fine/fast on servers.

Nice! Any way to subscribe for updates?

In European inner cities, it can be faster to get around on bikes.

I acknowledge Jevon and that demand may increase. But the original point was actually that specialized HW will be outcompeted by newer algorithms on more flexible HW. I have seen this play out several times, for example with brute force Sum of Abs Difference instructions that no video codec uses anymore, because a sparse search is faster than brute force HW. Personal opinion: this will happen again. Why should we focus on how to fill GPU cycles, rather than invent the future of how to do better?

Brute force hardware locks us into the local minimum of matmul. Stella Nera demonstrated better power efficiency than GPUs, using approx matmul. Maybe the next generation of special purpose HW can help, too. But they come after 2-4 years. Who knows what we will want then? I'd rather have flexible HW.

+1. Personal opinion: accelerators are useful today but have kept us in a local minimum which is certainly not the ideal. There are interesting approaches such as near linear low-rank approximation of attention gradients [1]. Would we rather have that, or somewhat better constant factors? [1] https://arxiv.org/html/2408.13233v1

Indeed, though Highway is approaching a system library at this point. It is also a thin wrapper over the compiler's intrinsics headers: e.g. ~16KLOC for arm_neon.h, times the number of targets. That's a lot of code, but it is not comparable with actual codec logic lines.

Highway TL here, happy to discuss :)

I generally agree with your points. On niceness of programming, our Highway SIMD library provides portable intrinsics that cover most of the nice SVE/RVV/AVX-512. Those intrinsics are reasonably emulated on other ISAs. Is it fair to say problem solved?

I do not understand this take :) The vast majority of our code is vector length agnostic, as is required for SVE and RVV.

GPU perf/TCO is not always better than CPU, if you can even get enough GPUs, and the latency is also a concern and inducement to do everything on the GPU, which can be limiting.

Going through Highway's set of shuffle ops:

64-bit OddEven/Reverse2/ConcatOdd/ConcatEven, OddEvenBlocks, SwapAdjacentBlocks, 8-bit Reverse, CombineShiftRightBytes, TableLookupBytesOr0 (=PSHUFB) and Broadcast especially for 8-bit, TwoTablesLookupLanes, InsertBlock, InterleaveLower/InterleaveUpper (=vzip1/2).

All of these are considerably more expensive on RVV. SVE has a nice set, despite also being VL-agnostic.

:) Yes indeed, no changes to the build required. Example: https://gcc.godbolt.org/z/KM3ben7ET

I agree dispatch should be at a reasonably high level. This hasn't been a problem in my experience, we have been able to inline together most SIMD code and dispatch infrequently.

Was thinking of a shorter avl producing partial results merged into another reg. Something like a += b; a[0] += c[0]. Without avl we'd just have a write-after-write, but with it, we now have an additional input, and whether this happens depends on global state (VL).

Espasa discusses this around 6:45 of https://www.youtube.com/watch?v=WzID6kk8RNs.

Agree agnostic would help, but the machine also has to handle SW asking for mask/tail unchanged, right?

> With 50+ years of figuring the basics out, RISC-V won't be making any major mistakes on the most important stuff.

RVV does have significant departures from prior work, and some of them are difficult to understand:

- the whole concept of avl, which adds complexity in many areas including reg renaming. From where I sit, we could just use masks instead.

- mask bits reside in the lower bits of a vector, so we either require tons of lane-crossing wires or some kind of caching.

- global state LMUL/SEW makes things hard for compilers and OoO.

- LMUL is cool but I imagine it's not fun to implement reductions, and vrgather.

> it's a matter of the wrong configuration unlike x86 where it is a matter of the wrong instruction

+1 to dzaima's mention of vrgather. The lack of fixed-pattern shuffle instructions in RVV is absolutely a wrong-instruction issue.

I agree with your point that multiple code variants + runtime dispatch are helpful. We do this with Highway in particular for x86. Users only write code once with portable intrinsics, and the mess of instruction selection is taken care of.

Cinebench might not be the most relevant benchmark, it uses lots of scalar instructions with fairly high branch mispredictions and low IPC: https://chipsandcheese.com/2021/02/22/analyzing-zen-2s-cineb....