Hacker News Reader

Comment by badsectoracula | Hacker News Reader

I've found this amusing because right now i'm downloading `wikipedia_en_all_maxi_2024-01.zim` so i can use it with an LLM with pages extracted using `libzim` :-P. AFAICT the zim files have the pages as HTML and the file i'm downloading is ~100GB.

(reason: trying to cross-reference my tons of downloaded games my HDD - for which i only have titles as i never bothered to do any further categorization over the years aside than the place i got them from - with wikipedia articles - assuming they have one - to organize them in genres, some info, etc and after some experimentation it turns out an LLM - specifically a quantized Mistral Small 3.2 - can make some sense of the chaos while being fast enough to run from scripts via a custom llama.cpp program)

View on HN

> trying to cross-reference my tons of downloaded games my HDD - for which i only have titles as i never bothered to do any further categorization over the years aside than the place i got them from - with wikipedia articles - assuming they have one - to organize them in genres, some info, etc and after some experimentation it turns out an LLM - specifically a quantized Mistral Small 3.2 - can make some sense of the chaos while being fast enough to run from scripts via a custom llama.cpp program

You can do this a lot easier with Wikidata queries, and that will also include known video games for which an English Wikipedia article doesn't exist yet.

I'm not sure about this, i just checked Tron 2.0 (just a random game i thought of) and Wikidata seems to have wrong info (e.g. genre) compared to the Wikipedia article. Also i need to it describe a bit with what the game is about since i want to generate an html file with all the games and do a quick scan of them and Wikidata doesn't have that.

IGDB would be a better source than Wikidata (especially since it does have a small description too) but i wanted to do things offline. And having Wikipedia locally doesn't hurt. And TBH i don't think it'd be any easier, extracting the data from Wikipedia pages was the most trivial part.

That said I'll need to use some other source at some point since, as you mentioned, Wikipedia does not have everything.

Would MobyGames be a better source for this information? The information is curated, and an API is available.

Now this is the juicy tidbits I read HN for! A proper comment about doing something technical with something that's been invested in personally in an interesting manner. With just enough detail to tantalise. This seems like the best use of GenAI so far. Not writing my code for me or helping me grock something I should just be reading the source for or pumping up a stupid start up funding grab. I've been working through building an LLM from scratch and this is one time it actually appears useful because for the life of me I just can't seem to find much value in it so far. I must have more to learn so thanks for the pointer.

Reply on news.ycombinator.com