I haven't taken the course, but that sounds like a horrible place to start a course on understanding deep learning. GPU matrix operations are literally an implementation detail.
I think the proper way to teach deep learning "from scratch" would be:
1.) show simple example of regression using high level library
2.) implement same regression by writing a simple neutral network from scratch (explain going through, multiplying weights, adding biases, applying activation function, calculating loss, back propagation).
3. use NN on more complicated problem with more parameters and a larger training set, so that user sees they've hit a wall in performance, and now implementation needs to be optimized
4. at this point, say okay, our implementation of looping and multiplying can be done much faster with matrix multiplication on GPU, and even faster parallelized across GPUs on a network. If you're interested in that, here is an optional fork in the course that gets into specifics. Anything after this point will assume that implementation of NN calls will be using these techniques under the hood
5. move onto classification, q learning, GANs, transformers.
95% should have skipped step 4 and only revisited if they become interested in this for a specific reason. To start with it is crazy. It's like starting a course about flying by explaining how certain composites allowed us to transition from propellers to jets, and let's dive into how those composites are made.