Hacker News Reader

Comment by jimmaswell | Hacker News Reader

A good way I found to store and use markov chain data was through the filesystem. Each words gets its own file, and every word that comes after that word is put in that file. The problem of the probability of one word coming after another solves itself if you just let the word be added to that list multiple times, one for each occurrence.

View on HN

But a file system forms a tree whereas a Markov chain is a directed graph, cycles allowed. Or are you thinking of a file system with cyclical symlinks?

As I understand it you would have, as an example (example as a cycle):

    file: As
    contains: I an a

    file: an
    contains: example

    file: example
    contains: example as

Here "as" points to "example", and "example" points to "example" and to "as". You have your cycle: pick a random file (get example) pick a random word (get as), lookup as (open the "as"-file), pick a random word (get "as" again, then "an" then "example" … ).

No links needed, the words are the links -- for US English, and words no longer that 8 characters, you could this on FAT16.

(The files are vertices, and they contain a list of edges to other vertices, in the form of the name of the vertices).

Yeah, that's how I had it. Newlines separating the words, and the chars that can't go in the filesystem get escaped to base64 or something.

Is it then possible to model 2nd order Markov chains with every two words containing their own file? Filename: this_example Contains: is, can, etc...

The way you described it sounds possible, yes

Reply on news.ycombinator.com