Directed Edge (wheels' startup) launches public beta, finds related articles in Wikipedia
blog.directededge.com
blog.directededge.com
I'm working with large graphs and I find that using a relational database as a backend is horribly slow if you want to run somewhat complex graph algorithms. Looks like everyone who does this ends up developing an in-house system. Anyone know if there's a library out there for doing this sort of thing? The order of magnitude I'm talking about is ~10^8 nodes, ~10^9 edges.
For real world problems, I tend to write custom code that's Java NIO heavy. Try to pack the data as efficiently as possible, and minimize disk seeks.
Also, while we're talking about Wikipedia-- any plans to play around with "shortest path" algorithms? For example, how few pages to I have to navigate to get from, say, "Turing machines" to "Miles Davis"?
you can find shortest paths here: http://www.netsoc.tcd.ie/~mu/wiki/
It also happens that I've done a bit of work in writing fast, embedded databases, so when I realized that none of the off-the-shelf graph databases that I found were fast enough for our needs, and that mapping graphs to an SQL database was abysmally slow, I wrote a small one. Honestly, I was surprised when I discovered some graph databases that advertise being an order of magnitude slower.
I'd love for our product to be a graph database, but I'm convinced that product takes too long to get to the market. I'd need probably a year of just concentrating on that, without, say, also having to do business stuff just to get that ready for general purpose use. It'd also be cooler for geeking out to, but I don't know if people would actually pay for it.
So right now we've got a database that's really fast at: traversing lots of edges and filtering on tags. That's what we need for finding related stuff.
As for shortest path stuff ... well, it's never had much more than novelty value for me. I don't see much of a business case for implementing it, and well, we're a business.
Personally, I like playing around with graph theory, but I have yet to come up with a compelling business case-- I wish you luck, and hope you find something cool (and lucrative.) I'd probably be a customer of a fast graph database, but I doubt there are a lot of other graph theory hobbyists out there.
And the related links feature makes ad-hoc browsing of wikipedia entries quite useful, again kudos.
We'll be adding other techniques to our analysis over time though, this is just our first step...
Why do I care, exactly? What's the value here?
My question would be, do you not associate what we're doing with recommendations or not see recommendations as valuable?
made along with HN user jsomers