Hacker News Reader

As far as APIs we only used the regular chat completion api and promoted the model to output a list of issues with a title and XPath.

The rest is leveraged by our existing chrome extension to capture the screenshot, show the pins, etc.

View on HN

Are you passing the entire context of the page code to the LLM to get back the XPath?

Yes, the whole document via document.documentElement.outerHTML. Surely there's a lot of optimizations we could look into but this was more to try if the model could localize elements via XPath by using screenshot + HTML. It's doing a decent job for the most part, I definitely have seen some XPath hallucinations though.

Reply on news.ycombinator.com