Beating CuBLAS is easy. I wrote a 30-line kernel that multiplies two 4096² matrices 14% faster than CuBLAS on a 4090.
The question is how to earn money on that.
On average over 20 runs:
CuBLAS (./sgemm 0) has 50.9 TFLOPS.
My kernel has 61.8 TFLOPS, so it's actually +21% speedup in this benchmark.
How do I collect my paycheck?
On a 4090 gpu, average of 20 runs of SGEMM_CUDA:
size tflops_cublas tflops_my diff
4096² 50.8-50.9 61.8 +21%
8192² 56.3-56.4 67.1 +19%
16384² 53.6 66.7 +24%
I have seen how those high-performance libraries are made and I'm still in awe at the quality and quantity of the staffing involved. Those were the smartest and most knowledgeable engineers I met in my career.
Generalizing from a micro benchmark is typically hubris.