juxtaposicion

How are these interactive visualizations made? As a senior machine learning engineer (with only rudimentary JS skills) it would be fantastically fun to make something like these.

Stitch Fix. Data Science Manager.

Come lead a team to build real-time state-of-the-art recommender systems.

See more here: https://docs.google.com/presentation/d/1xCUatmnolfzUzfHR6Wmq...

Apply here: https://www.stitchfix.com/careers/jobs?gh_jid=2211077&gh_jid...

How does this compare to Edward, PyMC, Stan, et al? Is the primary distinction due to PyTorch’s imperative, dynamic programming?

If you've seen tensor trains, might be interesting to reduce the rank even further by splitting an input into many sequential small matrices (instead of just two.)

Will Keras2 support PyTorch as backend, in the future?

Answer: [0] No, there are no plans to support PyTorch. There is nothing to be gained in supporting every novelty framework that crops up every quarter. Our goal is to make deep learning accessible and useful to as many people as possible, and that goal is completely opposite to building up deep learning hipster cred.

[0]: https://github.com/fchollet/keras/issues/5299

In general, TensorFlow and other automatic differentiation frameworks (Chainer, Torch, Theano, etc.) operate on gradient descent of smooth functions. If you can recast your combinatorial problem into a smooth function, then these frameworks may be applicable. However, most combinatorial optimization problems aren't readily transformed into smooth functions, so this is unlikely to be straightforward.

I implemented the new t-SNE in sklearn, so I've got some experience in reading these diagrams. Unfortunately, as wonderful as the algorithm is, it's extremely hard to interpret what it means rigorously. I've seen many diagrams that look like this one -- and they were generated from actual noise. So take the plots with a big grain of salt :)

I'd be interested in seeing more direct evidence, like SVD factorizing the PMI matrix (which is what similar to what word2vec is doing) and seeing how much of the variance is explained by the first components. If you want to do this, check out: https://minhlab.wordpress.com/2015/06/08/a-new-proof-for-the...

It combines the (arguably) best properties of both algorithms. Word2vec is local and creates word representations that are powerful and flexible. LDA is global, creating document representations that are less flexible but very interpretable to humans. lda2vec mixes both ideas.

Ultimately, the goal is to use all of the information that is usually available alongside text. Word2vec treats this text like one long string. LDA has the notion of documents. But lda2vec can use more features (for example) the zip code a client comment might come from (and so you get regional topics, like outer wear in Vermont or cowboy boots in Texas) the client ID a comment comes from (so you get that a client might be a sporty client, or a expecting mother) in addition to document-level topics (which might surface customer comments like "perfect service!" or package delivery problems). Those topics are readily consumed by analysts and can be used to understand the business from the client's perspective; word2vec on the other hand produces representations that are hard for anything but machines to consume.

I think you're right when thinking about CNNs on words. It's max-pool that's usually combined with CNNs that helps with translational invariance, less so the CNN filters themselves (which, if it were a full convolution, would show up in the complex phase).

I think it makes more sense when CNNs are applied at the character level. The filter banks then activate for specific n-gram patterns of characters, like certain prefixes, suffixes, and root words. The higher level LSTMs are then relieved of having to understand that level of structure. Also, tokenization is hard, and might be especially wrong for media with grammatical abuse like Twitter, and this avoids that janky preprocessing. See: http://arxiv.org/abs/1508.06615

The Stitch Fix Tech Blog is also nice: http://multithreaded.stitchfix.com/blog/

And has recently covered GAMs, mixed effects models, word2vec, and tools like Pyxley.

Thanks! The word 'vacation' is just removed from the list since it's exactly what we're looking for.

The item vectors are generated from the text: customers and stylists write text about the stripes, or maternity, and word2vec associates this with the item in question. How that happens is treated in the next section about summarizing documents (a 'document' here is the collection of all text about an item).

So we don't do any fancy deep learning from the images themselves, although this is on the horizon :)

It's my first time seeing the package, but looking over the docs it looks like it implements LSA. The major difference here is that word2vec dramatically outperforms LSA in a variety of tasks (http://datascience.stackexchange.com/questions/678/what-are-...). My experience has been that the vector representations in LSA can be underwhelming and poorly performant. I can't comment on the Random Projection and Reflective Random Indexing techniques SemanticVectors implements.

This link is about document distances but still compares other techniques nicely: http://datascience.stackexchange.com/questions/678/what-are-...

It's just a play on the phrase a 'picture is worth a thousand words' :) The word vectors themselves contain sophisticated relationships that seem almost miraculous, we're definitely not negative on them

(Speaking as one of the authors of the post)

In both this paper and word2vec, the key concept is to try and represent a high-dimensional and sparse dataset as a dense and low-dimensional continuous vector. Interestingly, the same skip-gram algorithm is used for both even though it's applied to datasets as disparate as a social graph and a sentence structure. There's a bit of cleverness here: the authors equate a sequence of social network graph visits (a random walk in DeepWalk) to a sequence of words (a sentence in word2vec.) In both cases the resulting representation is dense while still preserving many relevant properties of a social group which makes it useful as an input to other ML algorithms. Incredibly interesting.

I wonder if there's a simple but powerful example (like king-man+woman=queen for word2vec) of this technique.

Is Julia being used in production at StitchFix?

I hope this program grows. The thirst for high-quality data engineers is unquenchable.

This is particularly great for recommending new software for rich ecosystems like Hadoop. This highlighted the usual suspects: Hive, Pig, & Cassandra, but it also found a few Apache Incubation projects automagically:

http://inspectorgit.com/apache+hadoop-common

Found: Oozie, a 'worfklow' engine for Hadoop (https://github.com/yahoo/oozie) Netty, high-performance networking on top of Hadoop (https://github.com/netty/netty)

For what is similar to the linux kernel: http://inspectorgit.com/torvalds+linux

Gets Git itself as well memcached!

Thanks! I'll have to spend more time sanitizing the input.

Hey SandB0x, thanks for the advice! I actually started off by using numpy.dot, which is precisely what's needed. The problem is that I need it to go even faster (currently takes a few seconds) but this function is already heavily optimized and uses Intel MKL to accelerate the math. In fact, my cython implementation would be slower than numpy.dot were it not for some embedded logic that breaks out of the dot product half way through calculation. As I'm computing the row * row, going element by element, if the running sum of those products gets to be really negative (indicating that in the dimensions multiplied so far, the two rows are highly dissimilar) I stop tryin to calculate the rest of the row, thereby saving me from calculating the full dot product. This is cheating obvioisly, but it's x3 or x4 faster numpy.dot. So, because there's branching logic in my implementation of the dot product, I can't express it interns of Einstein summations.

Hey doctoboggan, awesome questions!

>> What is the dimensionality of each word vector and what does a words position in this space "mean"? What is this dimensionality determined by?

Each dimension roughly is a new way that words can be similar or dissimilar. So I've got 1000-dimensional vectors, so words can be similar or dissimilar in only one thousand 'ways'. So associations like 'luxury', 'thoughtful', 'person', 'place' or 'object' are learned (roughly speaking). Of course, real words are far more diverse, so this is an approximation. The 1000 dimensions is configurable, and in theory more dimensions means more contrast is captured, but you need more training data. In practice, the number 1000 is chosen because that maxes out the size of my large memory machine. That said, the word2vec paper shows good results with 1000D, so it doesn't seem to be a bad choice.

>> Have your tried any dimensionality reduction algorithms like PCA or Isomap?

Yes! I've tried out PCA, and some spectral biclustering using the off-the-shelf algorithms in SciKits Learn. I only played around with this for an hour or so but got discouraging results. Nevertheless, the word2vec papers actually show that this works really well for projecting France, USA, Paris, DC, London, etc. on a two-dimensional plane where the axis roughly correspond to countried & capitals -- exactly what you'd hope for! I wasn't able to replicate that, but Tomas Mikolov was!

>> It would be interesting to find the word vectors that contain the most variation across all of wikipedia.

Hmm, interesting indeed! I'm not sure how I'd got about measuring 'variation' -- would this amount to isolating word clusters and finding the most dense ones? Something like finding a cluster with a hundred variations of the word 'snow' (if you're Inuit)? I'd be willing to part with the raw vector database if there's interest.

>> Have you tried any other nearest neighbor search methods other than a simple dot product, such as locality sensitive hashing?

Only a little bit, although I'm very interested in finding a faster approach than finding the whole damn dot product (see: https://news.ycombinator.com/item?id=6720359). I worry that traditional location sensitive hashes, kd-trees, and the like work well for 3D locations, but miserably for 1000D data like I have here.

I should reiterate out that most of the hard work revolves around the word2vec algorithm which I used but didn't write. It's awesome, check it and the papers out here: https://code.google.com/p/word2vec/

Whoa, that was alot. Thanks!

Thanks! The number crunching is indeed very intense. The full body of vectors has a million rows and thousand columns, which fills up all of the 10GB of available memory. When you punch in a query, it adds or subtracts the requested vectors and takes an approximate dot product between every row in the table and the search query. This is about 10^9 operations. I'm only interested in the most similar (high cosine similarity) dot products, so to speed things up I wrote a Cython dot product implementation. This aborts the calculation if the sum starts to look like it'll be very dissimilar, essentially skipping lots of bad guesses. This speeds things up by a factor of ~5 or so. I'm debating offloading this computation to the GPU, which would be perfect for this.

Edit. In case you're interested in the source: https://github.com/cemoody/wizlang

Some of this is that underlying model is insufficiently trained, but some of this is disambiguation. Disambiguation in text is a very, very, hard problem.

So some of your examples, when clarified, are a bit clearer: Paul McCartney - Beatles + Rolling Stones = Mick Jagger is in the 3rd spot. (http://www.thisplusthat.me/search/Paul%20McCartney%20-%20Bea...) Change stone -> Rolling Stones.

Thank you for the comment!

It's not perfect; the real limiting factor is the volume of text. For the research paper behind word2vec, to get accurate associations for common words (King, man, woman, etc.) required a news sources training text of order a billion words. This is roughly the size of all of Wikipedia, but the text in Wikipedia has many more rare words, which spreads the number of training examples per word rather thin. To address that, I'm thinking of expanding the project to use Common Crawl data (http://commoncrawl.org/) to dramatically expand the available supply of data.

Hi, creator here (Chris Moody). Great question. The underlying algorithm, word2vec, (https://code.google.com/p/word2vec/) isn't built for streaming data which means that at the moment it assumes a fixed number of words from the beginning of the calculation. Unfortunately, until the state-of-the-art advances to accepting streaming data, the whole corpus will have to be rescanned to accept dynamic content. Furthermore, word2vec doesn't scale past OpenMP, single node, shared-memory resources. So while I used MapReduce, it's just for cleaning and preprocessing the text, not training the vectors, which is the hard part.

So there's some exciting work to be done in parallelizing and streaming the word2vec algorithm!

Harvard - Boston + Silicon http://www.thisplusthat.me/search/Harvard%20-%20Boston%20%2B...