Hacker News Reader

Comment by tom1337 | Hacker News Reader

I wondered how you are deciding on whether the response helped the user. Couldn't find it on your page quick so I asked your bot

> How are you checking whether the answer helped the user and that it needs to be billed? >I couldn't find a clear answer to that in the current docs.

So... how are you determining whether it helped the user? How are you ensuring that the model just doesn't make stuff up? Stripe has a similar feature where a LLM can help find the docs and it regularly hallucinates commands or arguments that just don't exist (but would perfectly solve my problem)

View on HN

I loop through all Q&As at the end of the billing period and ask gemini if this users query was solved.

Something fair for customers, thats the goal. But right now my priority is seeing if anyone gives a f** about this to begin with.

Regarding hallucinations: It's not 100% solvable, but can be done pretty well in my opinion with prompting (aka only answer based on facts you see in the documentation)

P.S. if you have docs and want a more "live demo" you can sign up and just put in the docs url and you'll have a live demo without installing the script onto the docs site.

Interesting approach using Gemini for checking whether the answer helped. I wonder how that would work if the response is hallucinated.

As an example: Stripe has sandboxes, I wanted to know how to create them via the CLI but found nothing in the docs. Asked the AI and it was like "Just do stripe sandboxes create <name>". The command does not exist and the whole functionality is not planned to be added. If another AI now reads this it could interpret it as answered successfully?

Just thinking - but I like the idea of the tool. Would love to use it but currently don't have any public docs to test with. Good luck with your project :)

--> I would say so. Not quite sure what Stripe is using on their docs, but sounds like its been given free reign and not grounded in actual docs.

Easiest way is to weed out hallucinations, hasnt been a problem for me at all to be honest.

P.S. For anyone reading, happy to refund all answers that you deem shouldnt be billed.

Just tested the tool on an open source documentation. Looks and feels great but somehow after the initial Preview in the onboarding flow I cannot get the Preview to work. It seems to always answer with the wrong dataset and suddenly says "The provided documents describe Agionic, an Al tool for documentation, and do not contain details about UI component libraries like Grid." Not sure whats happening here.

https://imgur.com/a/XiVmosl

I'm guessing the page was refreshed and the preview lost context to the right customer ID...

Thanks for trying it!

Reply on news.ycombinator.com