Comment by tgflynn | Hacker News Reader

Comment by tgflynn | Hacker News Reader

Parent Full thread

if you take some physically grounded data and change units, you won't recover different eigenvectors as long as your input data is good.

I don't think that's accurate. Consider a set of points distributed along a line in 2D. If you do PCA on these points you will find that the 1st eigenvector points along the line and the second is orthogonal to it.

If you now rescale the axes so that x is measured in meters and y in light years the slope of the line will change and so will the the 2 PCA eigenvectors.

However the relationship between the 2 eigenvectors and the distribution of the data points will remain the same. The first eigenvector will still point along the line and the second will still be orthogonal to it.

In machine learning one is interested in the distribution of the data not in whatever units they happen to be measured in, hence I don't understand MacKay's objection.

The mathematical way to state this is that a choice of units is equivalent to a choice of an inner product in linear algebra( or, more generally, the choice of a metric tensor in differential geometry). Basically, the choice of an inner product defines what it means for something to be orthogonal in a space.

Principal component analysis is predicated on a choice of inner product, since component directions are always chosen to be orthogonal. It's not clear what orthogonality could mean in a 2D plane where one direction is measured in inches and the other in tons, so naive PCA isn't appropriate in such a case.

Others have mentioned in this thread that "whitening" the data before PCA fixes this problem, by removing cross-correlations. Presumably, in that case, the notion of orthogonality is taken from the statistical properties of the data. (Maybe it normalizes physical units like inches to the standard deviation of the data's distribution in inches?)

But the principal eigenvectors can change if scales are changed - and the directions in which they may then point can be completely different.

Reply on news.ycombinator.com