This is interesting! I wonder how different the results are from just indexing the contents of the page and semantically searching them (vs. trying to predict the next link). Have you tried anything like that?
So metaphor is able to translate the language of comments (“here are some thoughtful, technical blog posts about AI”) to the language of documents.
Disclaimer: I also work on a semantic search engine.