This is really nice, especially for feeding LLMs web page data (they generally understand markdown well).
I built something similar for the Elixir world but it’s much more limited (I might borrow some of your ideas):
I built something similar for the Elixir world but it’s much more limited (I might borrow some of your ideas):
We interact with the web so much that it’s worth having such a library in every language... Great that you took the time and wrote one for the Elixir community!
Feel free to contact me if you want to ping-pong some ideas!
> feeding LLMs web page data
Exactly, that one use case that got quite popular. There is also the feature of keeping specific HTML tags (e.g. <article> and <footer>) to give the LLM a bit more context about the page.
Use https://tools.simonwillison.net/jina-reader to fetch the https://news.ycombinator.com/ homepage as Markdown and paste it into https://tools.simonwillison.net/claude-token-counter - 1550 tokens.
Same thing as HTML: 13367 tokens.
Possible approaches include transforming the text to MD or minimizing the HTML (e.g., removing script tags, comments, etc.).
I'd love to figure out a way to do it though, it seems to me that there's a bunch of rich description of the website in the html