Hacker News Reader

Comment by misiti3780 | Hacker News Reader

I see his conclusion is deep learning doesnt work here - but it might be interesting to incorporate word2vec features into this and see if the performance doesnt increase (or at least hover around 94%)

View on HN

I ran one experiment with an Embedding layer (second one in the Deep Learning Is No Good Here section), which I assumed to be analagous to word2vec. I hoped it would help as well so was rather dissapointed it did not seem to be so. Perhaps I should have tried a few more configurations, though.

An Embedding layer by itself will try to learn word vectors from the data you train it on (error gradient will propagate back to embedding layer, update the vector weights). Word2vec and word vectors are only really useful with tons of training data to learn good embeddings. I believe OP is referring to using Google's pre-trained word-vectors (trained on massive amounts of text [100 Billion words]). Can be found here: https://code.google.com/p/word2vec/

This is pretty straightforward to implement in keras, you just need to supply pre-trained word-vectors weights to your embedding layer.

  Embedding(vocab_size, 300, weights = [word2vec_weights])

Where 'word2vec_weights' is a numpy matrix with shape (vocab_size, 300).

Ah, I honestly assumed it would use an existing set of weights such as from word2vec. Though I now realize that does not make sense since you can also embed images or really any real numbered inputs. I will give this a try.

Cool! Let me know if you run into any issues.

Reply on news.ycombinator.com