Hacker News Reader

Comment by Radim | Hacker News Reader

The article seems a little rough (draft?), but the money shot is the algo in Section 5. Looks pretty neat and straightforward!

TL;DR:

1. Build a special distance(word1, word2) metric using a set of word vectors trained elsewhere (such as GloVe). This distance works better than cosine distance.

2. Given a document (a sentence, a paragraph… basically, a sequence of words), calculate the "importance" of each word as a sigmoid over distances(word, avg(all_words)).

3. To embed a document, simply do a weighted average of its word vectors, where the weight of each word equals the importance above.

View on HN

Hi, author here! I can confirm everything you said. A conference version is in the works, though this one is essentially complete

Do you plan to release an implementation? I would definitely like to play around with it.

Absolutely. The current priority is conference submission, but releasing a simple interface is definitely the plan, as appropriate

https://github.com/ezelikman/Context-Is-Everything The code will be posted here when available!

here's an impl of something like (but actually probably nothing like) 1. in the above tl;dr

(disclaimer: i dr)

https://github.com/ransomw/botiful-soup/blob/master/mibot/sr...

as i recall, there's a lot of neato REPL-y graphvis to try in

https://github.com/ransomw/botiful-soup/tree/master/mibot/de...

----

while i'm here, let me mention: Gärdenfors books like "Geometry of Meaning" are hearty food for thought about natural language semantics

A small suggestion: I think the way you're describing this as "context" might be misleading. You'll probably want to try the same thing on the output of a BiLSTM or CNN model instead of just the word vectors. I think this is more like an attention mechanism --- like tf-idf, you're learning to downweight words which are less informative. Your method is also very similar in effect to the LexRank line of work, which is more computationally expensive and complicated.

Hi! Can you clarify? This is a fairly standard use of "context" (http://www.socher.org/index.php/Main/ImprovingWordRepresenta...). While the algorithm can be used with just the covariance of a corpus's word vectors, accounting for the distribution of a context when setting up the distance metric acts to improve performance substantially (https://i.imgur.com/Dnx9c9G.jpg)

I just think that titling a paper about a weighted bag-of-words approach "Context is everything" really starts the reader off on the wrong foot. In fact there's almost no context in your model, in contrast to most other sentence learning methods!

To be clear, the weighted bag-of-words is a powerful application of the contextual salience measurement, CoSal, but is certainly not the only one.

(In practice, this can also be applied outside of language, for the importance of other semantic vectors in context)

‘Attention is all you need’, was the first thought i had when i read the title of your paper. It might be worthwhile to have a small bit explaining the distinction between attention and context as you view them. If i understand correctly you are describing how on to build a highly useful context representation. The application of that context in a neural net would be to focus the attention of the neural network on the parts of the input that are more important. From my point of view your paper here helps answer the question: what directs the attention?

Apologies if all of this is blindingly obvious. I’m a fascinated amateur.

Yep, exactly. The current paper presents the contextual model and how to interact with it, and then shows that even simple models (such as the weighted bag of words in the first parent comment) based on it do quite well. I really appreciate the interest and please let me know if you have any questions! :)

Was this a 224n project? I'm hoping to try doing the class online on my own. Do you think the 2017 videos are still pretty in-line with the topics on the current website?

Hi! This started off as a 224n project, but it's gone through quite a bit of work since then. The class slides are from the last class are pretty complete. I think they put a lot of time in to keep it updated, especially towards the end

Reply on news.ycombinator.com