And forgetting all the abstractions that make it work
If it was we'd have LLMs 20 years ago. Not even mnist is "just linear regression" (because neither sigmoid nor relu are linear)
And yes, that is indeed like saying a computer is just a bunch of zeroes and ones and logic gates. It's true and beautiful and profound but nearly useless from a practical perspective. And the wonder, like computers when you scale the numbers of transistors to the billions, is that when you scale the number of parameters to the billions, you end up with something amazing.
ReLu is truncated regression
It's all just GLMs!