Hacker News Reader

Comment by ContextArxiv | Hacker News Reader

Hi! Can you clarify? This is a fairly standard use of "context" (http://www.socher.org/index.php/Main/ImprovingWordRepresenta...). While the algorithm can be used with just the covariance of a corpus's word vectors, accounting for the distribution of a context when setting up the distance metric acts to improve performance substantially (https://i.imgur.com/Dnx9c9G.jpg)

View on HN

‘Attention is all you need’, was the first thought i had when i read the title of your paper. It might be worthwhile to have a small bit explaining the distinction between attention and context as you view them. If i understand correctly you are describing how on to build a highly useful context representation. The application of that context in a neural net would be to focus the attention of the neural network on the parts of the input that are more important. From my point of view your paper here helps answer the question: what directs the attention?

Apologies if all of this is blindingly obvious. I’m a fascinated amateur.

Yep, exactly. The current paper presents the contextual model and how to interact with it, and then shows that even simple models (such as the weighted bag of words in the first parent comment) based on it do quite well. I really appreciate the interest and please let me know if you have any questions! :)

I just think that titling a paper about a weighted bag-of-words approach "Context is everything" really starts the reader off on the wrong foot. In fact there's almost no context in your model, in contrast to most other sentence learning methods!

To be clear, the weighted bag-of-words is a powerful application of the contextual salience measurement, CoSal, but is certainly not the only one.

(In practice, this can also be applied outside of language, for the importance of other semantic vectors in context)

Reply on news.ycombinator.com