Hacker News Reader

Comment by doppenhe | Hacker News Reader

Agreed. Actually we could turn up the number of times it runs inside LDA algorithm and those would fix up but it affects performance. This was just a quick and dirty example (with an expectation of high traffic).

You can also seed LDA with a whitelist of words which we didn't do either - again all in the name of a quick and dirty solution to show.

Glad you liked it!

View on HN

How about the plain old traditional 'stop words' list to solve this issue?

Try using tf-idf instead of raw word frequencies.

not using raw word frequencies but http://en.wikipedia.org/wiki/Latent_Dirichlet_allocation.

Didnt know about tf-idf thanks for the tip.

it's hard to imagine someone knowing an LDA without also knowing about TF.IDF (it's a dot product, not a hyphen)

That and the questionable use of stopwords makes it sound like they're just slapping some marketing on an out-of-the box LDA implementation (not that I blame them, it's a dense algorithm).

The OP is pushing "Algorithmica" whose manifesto is here: http://blog.algorithmia.com/post/75680476188/algorithm-devel...

and this doesn't really strike me as much of a victory for the idea that it's just the implementation of an algorithm being the sticking point in practice.

We are just showing the versatility of the platform through a real world use case. LDA is hard to implement/scale for the untrained same as many other machine learning, optimization, graph traversing,etc algorithms. What we are building is crowd-sourced and generalized API where all these algorithms can be combined and used together to really make any application smarter.

The demo we show here is a version of how we used our platform to generate tags for all entries in our API by combining algorithms that existed already in Algorithmia. (modified for performance over quality due to the volume that HN would bring).

Cheers.

It's only hard because most libriaries I've seen have so little documentation available. It's simple once you understand the library. We need people picking these libraries up, implementing them on weekends on fun projects, documenting their work and code, and publishing it for everyone to learn from.

What about tag size (i.e., word length)? For the example of the knuth article, it'd be good if length was > 3.

I think you'd need to whitelist some useful acronyms if you implemented that rule.

USA

NSA

DOJ

TCP

POS

etc.

Reply on news.ycombinator.com