Hacker News Reader

You are asking exactly the right questions. As far as I know k-means will work well when you can answer those questions for your problem, otherwise not so much. In other words there's no silver bullet.

If you rephrase the "ML is easy" idea from the article to "it's easy to do some simple but cool things with ML" then it's true, but in pushing the envelope you can make it as complex as you like.

Great questions! I can answer a few of them.

- Yes, you can get stick in a local maximum. However, see the k-means++ algorithm for a clever initialization scheme that gets you within a good constant factor of the global maximum.

- The best way to pick K is to fit the algorithm with a range of K and see which K seems to give the best application performance.

- It is "your job" in creating the feature space to ensure that feature-space nearness corresponds to application-space nearness. This is not easy at all, and essentially the only reliable test if you've done a good job is if the final algorithm performs as you'd like it to.

I think that the post is really observing that, once you understand it, ML algorithms aren't really that complex. However, once you understand it, most things aren't that complex, and that doesn't make fully understanding them easy in the first place.

> This assumes that distance in the vector space strongly correlates to "similarity" in the thing I'm trying to understand. How do I know my vector model actually does that? (For example, how does the author know "has some word" is a useful metric for measuring post similarity?)

There isn't a great answer to this I'm afraid. It's a combination of "Well it seems to make sense" and "we tried it and it worked". There can be a lot of trial and error.

> (For example, how does the author know "has some word" is a useful metric for measuring post similarity?)

Specifically about this one, that's mostly because it makes some sense and is used a lot elsewhere. It's simple and tends to work reasonably well, doesn't require trying to parse and understand sentences, etc. The idea is that the words you say are likely to be closely linked to the thing you're talking about.

The next step up from this (usually called BoW, or Bag of Words) is fairly straightforward, it's how often each word appears as a fraction of the total document. Up from there is one of the most commonly used approaches called TF-IDF (term frequency inverse document frequency) which is almost the same but it modifies the counts based on how rare the words are. I go straight for TF-IDF as it's fairly sensible, simple and tends to work well in practice.

> I like what I got out of the post a lot, but the "this is easy" part only seems easy because it swept the hard part under the rug.

It's the difference between making something that works and making something that works well. A lot of domain knowledge and technical knowledge might be required for the second, but the first might be a lot more attainable than people think.

This is why the author of this post is wrong. The answer is that there is no known efficient algorithm to ensure you get to a maximum of the k-Means objective (and there is unlikely to be one), even if you fix the dimension of the vectors to two. Also, nobody knows how to pick K.