Agreed, there's something to be said for simple models that are "good enough," especially when their limitations are clear. k-NN also comes to mind.

Indeed. And e.g. generalized eigen problem extend this to the case of two competing data sets.

Ignoring the eigen aspect would miss a lot of both theory and practice.