WARC-GPT: An open-source tool for exploring web archives using AI | Hacker News Reader

WARC-GPT: An open-source tool for exploring web archives using AI

lil.law.harvard.edu

195 pointstoomuchtodo17 comments

If you just want semantic search, check out Semantra: https://github.com/freedmand/semantra

Nice, been looking for something like this. In case you - like me - are wondering on the embedding models options: https://github.com/freedmand/semantra/blob/main/docs/guide_m...

Looking into this, could be very useful for working with personal Zotero archives of webpages

As previously seen on HN:

https://github.com/Frost-group/The-Oracle-of-Zotero

There’s also ARIA [1]. I actually think the one linked above seems more interesting. At the same time, doesn’t seem all that different from just rolling your own solution.

[1] https://github.com/lifan0127/ai-research-assistant

paper-qa: https://github.com/whitead/paper-qa :

> LLM Chain querying documents with citations [e.g. a scientific Zotero library]

> This is a minimal package for doing question and answering from PDFs or text files (which can be raw HTML). It strives to give very good answers, with no hallucinations, by grounding responses with in-text citations.

  pip install paper-qa

> If you use Zotero to organize your personal bibliography, you can use the paperqa.contrib.ZoteroDB to query papers from your library, which relies on pyzotero. Install pyzotero to use this feature:

  pip install pyzotero

> If you want to use [ paperqa with pyzotero ] in an jupyter notebook or colab, you need to run the following command:

  import nest_asyncio
  nest_asyncio.apply()

... there is also neuml/paper-ai w/ paperetl; https://news.ycombinator.com/item?id=39363115 : gh topic: pdfgpt, chatpdf,

neuml/paperai has a YAML report definition schema: https://github.com/neuml/paperai :

> Semantic search and workflows for medical/scientific paper

  python -m paperai.report report.yml 50 md <path to model directory>

> The following [columns and answers] report [output] formats are supported: [Markdown, CSV], Annotation - Columns and answers are extracted from articles with the results annotated over the original PDF files. Requires passing in a path with the original PDF files.

Any nice tool to archive web sites using the warc format?

The Wikipedia article on the warc format references wget

May be some here https://github.com/iipc/awesome-web-archiving/blob/main/READ...

https://github.com/ArchiveTeam/grab-site

https://wiki.archiveteam.org/index.php/The_WARC_Ecosystem

If you want a full fidelity warc file, browsertrix-crawler is nice. It is slower than wget on account of using chrome, but it works better for sites with highly dynamic content and can generate wacz files which can be efficiently served via a file/objectstore when used with something like replayweb.page.

Webrecorder's are by far the best imo. https://archiveweb.page and Browsertrix Crawler. ArchiveBox uses wget internally for it's WARC generation but I'd love to integrate it with Webrecorder in the future for this part.

Is this something that could be used to create a chatbot with a knowledge base based on all pages and all snapshots of those pages of a specific domain archived by web.archive.org/?

> it is not meant to replace keyword and filter-based search, but instead to complement it, providing a different starting point for scholarly inquiry and potentially expanding access to otherwise hard-to-reach information.

I first read it as "War-GPT" and was very relieved to find it was otherwise.

WOPR-GPT

web 4.0?

Reply on news.ycombinator.com