Hacker News Reader

What open source model is good at pulling keyterms?

OpenNRE (https://github.com/thunlp/OpenNRE) is another good approach to neural relation extraction, though it's slightly dated. What would be particularly interesting is to combine models like OpenNRE or SpanMarker with entity-linking models to construct KG triples. And a solid, scalable graph database underneath would make for a great knowledge base that can be constructed from unstructured text.

Nice, I’ll look that up.

I was thinking in terms of RAG and turning text into keywords. Any thoughts there?

By this I presume you mean build a search index that can retrieve results based on keywords? I know certain databases use Lucene to build a keyword-based index on top of unstructured blobs of data. Another alternative is to use Tantivy (https://github.com/quickwit-oss/tantivy), a Rust version of Lucene, if building search indices via Java isn't your cup of tea :)

Both libraries offer multilingual support for keywords, I believe, so that's a benefit to vector search where multilingual embedding models are rather expensive.

For entity extraction you can look at SpanMarker: https://tomaarsen.github.io/SpanMarkerNER/. I'm sure other tools exists and others can hopefully point at more.

Reply on news.ycombinator.com