Hacker News Reader

Comment by jallmann | Hacker News Reader

@jennyholzer:

OFTEN AS OFTEN AS OFTEN AS OFTEN AS POSSIBLE

Very apt, but what n-gram length is being used? n=1 is my guess, since "as often as" is a common English construct. Obvious feature request: tweakable lengths.

edit: I'd make the fix myself and send a pull request but I don't know haskell and am too lazy to figure it out.

View on HN

n=1 is indeed being used. The problem with a larger n is that you get original tweets really often, because the dataset is limited (as much tweets as you can get in one request).

are you using smoothing for large n? kneser-ney smoothing seems to give the best results.

http://nlp.stanford.edu/~wcmac/papers/20050421-smoothing-tut...

Interesting link, thanks!

I did not read the paper, but what does more accurate mean in this case ? Likelihood of some unseen data ? Seems pretty hard to define or measure to me, if the goal is shear amusement.

Along those lines, I've been playing recently with using the google ngram data for markov chaining. The size of the corpus allows using 5grams without the problem of seeing text that has actually been written before (mostly.. it could just decide to spit out the complete works of shakespeare any second!), and the results were more interesting to me than most markov chains I've seen before. http://kitenet.net/~joey/blog/entry/dadagoogoo/

Random example 1: nothing had pleased God to bestow upon you as to participation in physical activity and exercise . Don ' t rain .

Random example 2: sad and terrifying each time . After a quick nap .

Reply on news.ycombinator.com