> Matrix multiplication is an embarrassingly parallel task.
(GPU programmer take). This is a common misconception. In reality, matrix multiplication is _extremely tricky_ to implement a fast parallel algorithm for because the number of useful instructions you can perform per byte is limited and you end up getting memory bound very quickly, so you start needing to think about properly employing LDS, managing bank conflicts, etc. For many applications that actually need to run fast, matrices are great for sorting out the theory, but then it's a good time to come up with something that will actually run fast after the paper problem is sorted out.