Ask HN: how to generate recommendations on the fly?
I really want to dive into this world (of generating recommendations), and I wonder if some people would be willing to share practical advice. I have seen some theoretical algorithms, but they seem unlikely to perform well with classical databases. Also, the books usually don't mention how to handle incremental growth of the data set. Suppose I use a standard k-means clustering algorithm, it seems unreasonable to shift thousands of nodes around just because of one new entry (and with a classical db, that would be a lot of writes).
I almost suspect there is no way around creating custom solutions - like attempting to keep useful parts of the data in memory. For inserting new entries, I suppose the existing nodes could be updated by and by. I hope that at least some of the algorithms react benign to being twisted (for example in k-means there is a lot of variability in how to define distance and attraction).
Does a graph database like neo4J even help much? Are databases like CouchDB usable?