No, it's not a linear regression, but it is at its heart optimization.

And yes, that is indeed like saying a computer is just a bunch of zeroes and ones and logic gates. It's true and beautiful and profound but nearly useless from a practical perspective. And the wonder, like computers when you scale the numbers of transistors to the billions, is that when you scale the number of parameters to the billions, you end up with something amazing.