Comment by rspeer | Hacker News Reader

Comment by rspeer | Hacker News Reader

Parent Full thread

Words with similar vectors have similar meanings. You use this in search, sentiment analysis, topic detection, finding similar text, and classification.

Of course there are tests for this "words with similar vectors have similar meanings" property, and I'm finding that the fastText vectors aren't doing that well on them, especially outside of English.

I'm glad they released them, particularly so anyone can run a fair comparison between different word vectors, but these things should come with evaluation results the same way that code should come with tests. These vectors are performing worse than ones that my company Luminoso released last year [1] (a better, later post is [2]), and if you don't believe me plugging my own vectors, I know that Sapienza University of Rome also has better vectors called NASARI [3].

fastText covers more languages, but most of these languages have no evaluations. How do you know the Basque vectors aren't just random numbers?

I think that performance hits a plateau when the vectors only come from text, with no relational knowledge, and especially when that text is only from Wikipedia. Text exists that isn't written like an encyclopedia. Meanings exist that aren't obvious from context. My research at Luminoso involves adding information from the ConceptNet knowledge graph, producing a word vector set called "ConceptNet Numberbatch" that just won against other systems in SemEval [4], a simultaneous, blind evaluation. The NASARI vectors are also based on a knowledge graph.

[1] https://blog.conceptnet.io/2016/05/19/an-introduction-to-the... -- linking this to establish the date

[2] https://blog.conceptnet.io/2016/11/03/conceptnet-5-5-and-con...

[3] http://lcl.uniroma1.it/nasari/

[4] http://alt.qcri.org/semeval2017/task2/

We evaluated using ConceptNet Numberbatch but in the end went with fasttext because of the treatment of OOV words using sub word information. This is important for us because we work with Social Media where misspellings are very frequent and we have found this helps a lot. Are you also looking into these sort of enhancements? How do you usually deal with OOV words?

I googled that and got

  Showing results for *conceptnet cumberbatch*

Oh, the many names of Bumblebee Banglesnatch.

Very good question!

Our OOV strategy was pretty important in SemEval. The first line of defense -- so fundamental to Numberbatch that I don't even think of it as OOV -- is to see if the term exists in ConceptNet but with too low a degree to make it into the matrix. In that case, we average the vectors from its neighbors in the graph that are in the matrix.

For handling words that are truly OOV for ConceptNet, we ended up using a simple strategy of matching prefixes of the word against known words (and also checking whether a word that's supposed to be in a different language was known in English).

fastText's sub-word strategy, which is learned along with the vocabulary instead of after the fact, is indeed a benefit they have. But am I right that the sub-word information isn't present in these vectors they released?

There's a paper on the SemEval results that just needs to be reviewed by the other participants, and I'm also working on a blog update about it.

> Words with similar vectors have similar meanings.

Can you explain WHY this is the case with word2vec? I have come across a paper which says "we don't know really". Is this true?

https://arxiv.org/abs/1402.3722

I think there have been some attempts to explain, none of them fully satisfying.

But there have also been systems that create vectors with that property more directly instead of as a side-effect of a neural net. Examples include Stanford's GloVe, or Omer Levy's Hyperwords, which is made entirely from old-school ideas such as mutual information and dimensionality reduction, and was for a while the best system if you limited it to fixed training data (still Wikipedia).

A gripe: reviewers don't even seem to like explanations. If you can explain your system with well-understood operations, it's boring and "not novel". But when Google publishes magical mystery vectors, they lap it up.

Reply on news.ycombinator.com