Hacker News Reader

Comment by SilasX | Hacker News Reader

So that's the result? That you can find sorta clever vector equations in the results like "Obama + Russia - USA = Putin"?

View on HN

Yes.

There is some nifty work on trying to structure an actual grammar based on the geometry of the embedding space, but "look at this cool thing we found" is a totally worthwhile thing to publish.

(Embeddings themselves are useful for a whole lot more than that, though.)

If these are the most interesting results, then it's not. Most of the work seems to be in the (human provided) search for happy combinations like these. Are they typical? Is the whole space like this? Is there a consistent metric for deciding that these are in fact clever and it's not just the human doing the work in making it make sense in a funny way?

As pointed out elsewhere, the results are pretty accurate, not nearly perfect but enough to open up new applications. http://arxiv.org/pdf/1301.3781.pdf

Word2vec is exciting because it has a useable amount of accuracy for a relatively small amount of cleverness on the part of the human. You can just throw a huge corpus at it and it performs reasonably well.

What does "accurate" mean in this context? If there's a standard meaning of it, I'd be glad to judge it by that.

But the metric the author offered, as their defining results, was some cute things with Putin and Obama.

So yeah, if the results are accurate relative to some standard metric, I wasn't disputing that. I was questioning the reliability of the results the author presented, about cute Obama/Putin connections.

It's not just "cute", that's the whole point.

Somewhat surprisingly, it was found that similarity of word representations goes beyond simple syntactic regularities. Using a word offset technique where simple algebraic operations are performed on the word vectors, it was shown for example that

  vector(”King”) - vector(”Man”) + vector(”Woman”)

results in a vector that is closest to the vector representation of the word Queen. In this paper, we try to maximize accuracy of these vector operations by developing new model architectures that preserve the linear regularities among words. We design a new comprehensive test set for measuring both syntactic and semantic regularities, and show that many such regularities can be learned with high accuracy.

And if the author highlighted those as "the results", I would be impressed. As it stands, they were citing more "moonshot" kind of results like the ability to get abstract concept composition right, which is really hard to judge, because you don't know what set to sample over, or whether the equations are really insightful vs "I guess that might make sense...".

I'm much more confused over how the author characterized the results ("look at this Animal/Human/Ethics thing!"). Again, if there are standard metrics by which it did well, great! But why isn't he citing them rather than these more abstract, hard-to-judge victories?

This is the exciting part - where it feels like the computer understands you. It's not a test, it's just a conversation. You can ask the computer a question and get a really meaningful answer.

... So long as it's not just "Eliza" meaningful.

As stated above in the thread, it has a 50% accuracy rate on an analogy dataset.

More significant I guess is how excited the author manges to be over these coincidences. Maybe a word of caution is needed: Overly generous interpretations is how things like Nostradamus or the hidden code of the bible retain their credibility with parts of the population.

And frankly, if you think that stock market \approx thermometer is insightful, you should probably be kept away from positions of responsibility.

Try some yourself. http://radimrehurek.com/2014/02/word2vec-tutorial/

Scroll down and you have some text boxes you can try some yourself. Here's some I tried (filtering out repeats and plural forms of input words. Those artifacts seem to happen a lot and would be easy to ignore)

cat:dog::bread:butter eh? I guess.

sword:shield::attack:protect okay that works.

up:down::left:leaving Eh, not great. I guess if you think they're analogous in terms of tense it kind of works. Disappointing, word2vec. (to be fair, "right" was third highest)

drive:car::dick:? Whatever it was, it made me giggle immaturely.

I bet you could run a bruteforce search over the possibility space for equations that can be made from simple vector addition/subtraction operations within some margin of error (since nothing perfectly overlaps) and look for interesting candidates manually post-hoc.

Really, there's a lot of \approx going on here -- you have thresholds for distance inside your vector space. Not only that, this shows that he cherrypicked "great" examples.

Reply on news.ycombinator.com