Something of a nitpick, but one thing that both Simon Ward-Jones and Joel Grus miss is that linear regression is typically not implemented using gradient descent at all, there's an analytical solution and you can get the beta coefficients with straightforward matrix algebra. It's much harder to explain than gradient descent so I get why they don't bother, but on the other hand without that background it's hard to see why everybody talks about linear regression all the time when with gradient descent or any other numerical optimizer there's really no limit to what f(x) can look like.