It always seemed to me like SIMD/AVX/etc would eventually come for the GPU's lunch money... How many more product generations of "SIMD on steroids" before this is practically true?
The latency factor is the biggest thing for me. The GPU is a turtle compared to CPU-bound techniques. I can see emerging applications for this in real-time/streaming where every millisecond counts.
GPUs are an optimization to try to use the excess Moore's law we have to get to the ghost of Dennard's law.
You also forgot to mention the usability bit. Branching is bad for GPUs. They either take both branches or in the optimized version, they stall the pipeline until they get a result then attempt to optimize which units need to execute on which branch.
The first is simply inefficient. The last is efficient, but impracticality slow on branchy code.
On the power side, bloated x86 is hardly a good showcase for what such an architecture could do. I also suspect that an automatic thread manager like most modern GPUs use could also add a lot of value.
Branching is bad for wide SIMD workloads of which GPUs are the most common. But just shoving that SIMD under the control of a CPU instead won't fix your branching issues. You're still probably going to choose the path of just executing both branches & masking off the results.
I seem to remember that LRB could test & jmp in 4 cycles, if you were careful. Since it was 4x barrel processed, those jmps were “free”.
I moved to integrated GPUs, later. x86 is bloated, but LRB was not. Also, the decoder — even a big one like x86 — isn’t a major HW problem. I’d say x86’s memory hierarchy is more of an issue.
Building something that can reliably output reasonable-quality 3d graphics without relying on specific GPU technologies will give you a much broader realm to operate with.
I believe something along this path is the solution for streaming gaming. I perceive the failure of Stadia, et. al. as being a consequence of trying to bolt streaming onto existing GPU-based, local gaming solutions. Build something from scratch with streaming/latency as a #1 priority, and you can dramatically expand the operational radius of each datacenter (e.g. ~100km per millisecond saved).
NVIDIA's streaming service is doing relatively fine in comparison. They simply share a GPU between several users for anything that isn't demanding enough. They also get around some of the concerns about gaming being turned into another streaming style fragmented mess by not actually selling the games. You simply log into your account on Steam/GOG/whatever and play the games you already own as you might on a local PC.
Additionally, "building something that can reliably output reasonable-quality 3d graphics without relying on specific GPU technologies" doesn't make much sense to me. If it's an accelerator designed to handle relatively modern 3d graphics, due to the programmability of a modern graphics pipeline it's effectively just a GPU. There aren't any underlying technologies that are required to be used as long as they can produce a similar output (mobile GPUs tend to have a different approach to how they implement the graphics pipeline compared to desktop GPUs for instance).
If you're talking full-screen Angry Birds with, say, a 2x average compositing, you're going to be fine on the CPU; but, energy- and jitter- wise you'll still be happier with the GPU, overall.
1) 1.5 TFLOPS is already less than the GPUs in most current phones. Like you're not exactly talking a big number here. You're talking 14 year old graphics (when desktop GPUs crossed the 1TFLOP mark)
2) And, this is the bigger issue, this AMX unit only does matrix multiplies. I'd be fascinated to see someone create a 3d renderer with only matrix multiplication. I'm sure it's possible but this 1.5 tflops ain't remotely comparable to a GPU's 1.5 tflops. Like if you tried to do "traditional" pixel shader rendering code on this unit you'd instantly be cutting that 1.5 tflops to 1/16th the performance - you're now at a theoretical peak of a mere 93 gflops of GPU-comparable performance.
https://www.pingdom.com/blog/theoretical-vs-real-world-speed...: “you should probably double the “ideal” response times shown above for a more realistic target to aim at“
So yes, ⅓ of light speed in vacuum seems a decent heuristic.
And also, as weird as it is, 1.5TFlops isn't actually that ridiculous. We had that performance 14 years ago at 150w with desktop GPUs. 14 years to reduce from 150w to what, 5w?, is cool but also honestly pretty par for the course is it not? Especially for a fixed-function block?
I don't want to give up SO-DIMMs for a few mm thinner laptop, but going from the intel/amd standard 70GB/sec to 400GB/sec is a pretty big incentive.
Regardless, no that's not the drastic shift necessary. It's not even a shift at all. 4, 6, and 8 channel memory architectures have all existed for a while now in data center SKUs. It hasn't changed the desire for GPU compute in the data center
Well the 4, 6, and 8 channel memory architectures have been dramatically slower than the Apple 800GB/sec with dramatically less memory transactions in flight (1 per channel) than Apple's 800GB/sec with 64 channels. The push for Apple like memory systems with HBM from Intel, AMD, Fujitsu and others does show that there's a need for more compute bandwidth and Fujitsu in particular has shown that a healthy CPU with a GPU like memory system can get great performance on real world codes without the hassle of rewriting for CUDA.
With all that said, yes there's a healthy demand for GPUs and codes that run best on GPUs. However I find it refreshing that at last one company is providing significantly improved memory systems in laptops and small desktops.
"the M1 Max isn’t able to fully saturate the SoC bandwidth from just the CPU side; From a single core perspective [..] it’s able to stress the memory fabric to up to 102GB/s. [..] Adding a third thread there’s a bit of an imbalance across the clusters, DRAM bandwidth goes to 204GB/s, but a fourth thread lands us at 224GB/s and this appears to be the limit on the SoC fabric that the CPUs are able to achieve"
https://www.anandtech.com/show/17024/apple-m1-max-performanc...
So peak CPU bandwidth on the M1 Max is 224GB/s. Which is really good for a CPU to be sure, but we're still talking bandwidth numbers far below what you'd need to eliminate the GPU.
> Well the 4, 6, and 8 channel memory architectures have been dramatically slower than the Apple 800GB/sec with dramatically less memory transactions in flight (1 per channel)
Huh? I guarantee you Xeons and Epycs are not limited to 1 transaction in flight, nor is Graviton2.
The AMD APUs in things like the steam deck, PS4, and PS5 (448GB/s fwiw) are also definitely not limited like that, either.
> than Apple's 800GB/sec with 64 channels
M1 ultra is 32 channels. Well, it's really 2x 16 channels since it's a NUMA configuration, like a dual-socket system.
> The push for Apple like memory systems with HBM from Intel, AMD, Fujitsu
lol? Apple's memory system isn't like HBM. It's more like a traditional GPU's but with GDDR swapped out for LPDDR. M1 Max is 512-bit memory bus, just like many GPUs have done over the years. 512-bit bus and it can only hit 400GB/s shows you just how much less bandwidth LPDDR has vs. GDDR, too. AMD & NVidia are hitting 1TB/s with 384-bit buses these days. Which is also what the HBM2e Xeon's are claiming.
Sure, wouldn't want to starve the GPU when the CPU is busy. Not sure if the AMX has it's own connection or shares the CPU complex bandwidth.
> I guarantee you Xeons and Epycs are not limited to 1 transaction in flight, nor is Graviton2
Right, I said 1 per channel, not 1. I've tested this on Xeons and Epycs, but not genoa yet. Generally maximum random throughput tends to be with 16 or so cache misses per socket with 8 channels. From what I can tell about half the latency is cache misses in L1, L2, and L3. Then you queue in the memory controller and wait for whatever memory channel (or two depending on config). By keeping 16 in the queue as soon as one of the 8 channels free you have another request pending for that channel, at least most of the time.
The Genoa looks particularly promising on this front, they upgraded from 8 channels to 12, but the DDR5 dimms actually provide two channels per dimm, so you end up with 24 narrow (32 bit) DDR5 channels. I've already seen results showing that Genoa scales better to high core counts than the zen3 Epyycs, which makes sense with 24 memory requests in flight instead of 8.
> M1 ultra is 32 channels. Well, it's really 2x 16 channels since it's a NUMA configuration, like a dual-socket system.
Heh, well monolithic chips are on the way out. Even the zen2 epycs used multiple pieces of silicon (chiplets) and the new Intel Xeon/Sapphire Rapids do the same. Don't see a particular difference. Even the desktop Ryzens have multiple pieces of silicon (one IOD and one CPU chiplet at the minimum). In any case the Apple M1 Max memory system matches the Genoa and beats the newest Intel Sapphire Rapids with a single piece of silicon, and wins handily doubles that with two pieces of silicon, unless you wait for the unreleased HBM flavors.
Compared to 100% of Intel/AMD laptops and probably 98% of desktops that apple memory system is pretty amazing. Sure there are threadrippers (4 channel), threadripper pro (8 channel), and some similar Intel options, but all are expensive, low volume, and desktop/workstation only.
Ok, then surely you won't take issue with me describing my x86 desktop as having 1.2TB/s of memory bandwidth? It just happens to be in a NUMA configuration and after all, wouldn't want to starve the GPU when the CPU is busy...
> Don't see a particular difference.
You don't see a difference between 1 memory controller and 2 memory controllers? Compare Zen1 TR/epyc vs. zen2 TR/epyc.
Hint: m1 ultra is like Zen1, and probably ain't the future
Oh yeah and Intel did this, too, with the 56 core xeons a couple years ago. And, like AMD, also quickly moved away from it.
Unless it's a rather odd config that's counting the GPU memory bandwidth which is decidedly not NUMA since the GPU mem isn't cache coherent with the main memory.
> You don't see a difference between 1 memory controller and 2 memory controllers?
From what I can tell the multiple memory controllers per chip AMDs performed well on most codes, but certain codes, OSs, and benchmarks didn't handle the different latencies within the socket well. As a result AMD moved to one memory controller per socket in the next generation.
The M1 ultra moved to two chips and 2 memory controllers, but as a result manages to hit 800GB/sec peak which is a pretty big win compared to any normal socket, even the 96 core AMD Genoa or Intel SPR are sitting in the half to 1/3rd range, despite coming our a year or so later.
It makes some GPGPU workloads viable that otherwise wouldn't be, but it doesn't reduce the bandwidth needs of the GPU for traditional GPU workloads so net-net you're worse off overall. You either use DDR and penalize your GPU performance, or you use GDDR (like consoles do) and sacrifice CPU memory latency. Also power efficiency of GDDR is worse, especially compared to Apple Silicon's choice of using LPDDR
HBM2e gets you everything except it's expensive and limited capacity. But it'll be interesting to see how that plays out with the Xeon Max, which can also still supplement the 64GB HBM2e with some absurd channel count DDR5
But otherwise that's less than what mobile GPUs can do, although those take more power they're not in entirely different ballparks either
That said I think the “good enough” metric is already there and unless you’re doing hardware ray tracing or extreme details at high resolutions you won’t need or care about a GPU any more.
Latency though isn’t the issue. The times involved for human perception are long and not getting shorter.
I'm not sure GPUs will be displaced, looking at the difficulties Larrabe had on the driver side, but I do think we'll see more flexible alternatives becoming popular.