Good to see GPT-2 done with shader programming. A port of the techniques used in smaller models, like TinyLlama or Gemma-2B, might lead to more experimentation with older or cheaper hardware [on-site].
NVidia has a lot to thank the competition why they happen to dominate the market.
I agree the competition mostly did it to themselves. I'll add Nvidia throwing GPU's at academics who built a lot of stuff on them. Classic strategy that goes back to IBM vs Burroughs.
OpenCL has been pretty handy for inference on older cards, but I'd argue it's relevance is waning. llama.cpp has Vulkan compute now which requires a smaller featureset for hardware to support. Many consumer devices skip OpenCL/CUDA altogether and delegate inference to an NPU library.
My Jax models and the baseline PyTorch models were quite easy to set up there, and there was not a noticeable perf difference to 8x A100s (which I used for prototyping on our university cluster) in practice.
Of course it’s just a random anecdote, but I don’t think nvidia is actually that much ahead.
They'd probably have to spend $5k-20k on a multicore or NUMA-style box to get huge gains on multithreaded code. They also loose the cool factor of saying they're using a RTX. Maybe grant money if it's tied to GPU use. Between the three, it might make sense, even financial sense, to get a sub-$2000 GPU to accelerate academic code that barely uses the GPU.
I'm just brainstorming here, though.
How many sub-$100 cards could train and inference with OpenCL vs Vulkan? Which is the bigger opportunity for poor people right now if doing GPGPU projects on local hardware?