Hacker News Reader

Comment by habitue | Hacker News Reader

Supervised disambiguation? Isn't the entire reason word2vec is exciting because it's unsupervised?

I have seen this many times that people claim the word2vec is unsupervised. But I think that is inaccurate.

word2vec is using a very weak supervision which is the order in which words appear in a meaningful sentence. And I think it is fascinating to use this kind of weak supervision to build distributed embedding for words.

This makes me wonder if the supervised/unsupervised distinction is useful at all. Maybe you could say instead that all learning algorithms simply need some way to measure similarity between the training examples, and it doesn't make a fundamental difference if you cluster examples by their target label ("supervised") or by their input vectors ("unsupervised") or by the context in which they appear (word2vec).

Skip-grams only take focus word, context word pairs. So those pairs do not take order into account

being in the same context is some kind of order.

That's still a bit far fetched. "Weakly supervised" refers to using a small amount of labeled data. This is not the case for word2vec and similar embedding methods. As a matter of fact, rather, the method presented here, sense2vec, would qualify, at it indeed is weakly supervised.

Where we use "small amount of labeled data" is called "semi-supervised".

> word2vec is using a very weak supervision which is the order in which words appear in a meaningful sentence.

What's supervised about that? Can't this be produced by a large enough corpus?

Not the only reason though. Its exciting because it turns a word into a reasonably sized vector. After that the vector can be fed into neural networks (nueral networks take vectors for input) or operations can be performed. Like Paris - France ~= Beijing - China.

Word2Vec is exciting because of its result, not particularly its method. This paper is a pretty straight forward extension of word2vec by combining it's 'unsupervised' result with various labels to make it more useful. You can think of word2vec as a kind of independent representation of language (which is a stretch for the purists but for sake of conversation I think it's okay) that can be applied to various domains by 'supervising' it.

Reply on news.ycombinator.com