GPT3 Get answers to technical questions from your documentation site | Hacker News Reader

GPT3 Get answers to technical questions from your documentation site

jointwig.com

Waiting lists and signup pages can't be Show HNs. Please see https://news.ycombinator.com/showhn.html.

I've taken Show HN out of the title now.

Dang, Hi, If you see just below the "add me to waitlist", There are live Chat Bots created for several companies you can try out right now. These chat interfaces are live and work on custom/fine-tuned models that have been trained on the documentation of these companies.

Over the last 9hrs users exchanged over 6000 messages with the bots. So it's live, it works, and users can try and play with it right now and give feedback.

A few hrs ago the bots hit OpenAis usage limits, Else its was working actively with over several hundred unique users interacting with the bot asking 4-5 questions each

maybe if you put the waitlist signup at the bottom/it only shows up after some usage?

I think the actual interface with OpenAI’s platform is the easy part. Everybody and their dog will have a version of this. Just look at the comments so far - many of us have already been playing with it.

If you want a real moat, figure out how to parse existing PDF documentation that is really badly formatted. Think diagrams and tables with text floating in various places, etc. Documentation of this style is very common in industries where physical things are being built in the real world. The standards documentation (IEEE, ANSI, NFPA, etc) doesn’t usually parse cleanly, much less the messier internal documentation within the businesses.

Grobid is the best example of such a documentation parser, but it is so laser focused on academic papers that it fails to properly process industry-style standards and SOP documentation. What the world needs right now is a Grobid that works for other kinds of messy documentation.

I think you are right, this will be the key differentiator for anyone building a service like this - I guess like with most machine learning/data science projects - the real work is on the data engineering side of things.

One thing that all these models will lack is the ability to include diagrams (on both the input and output side). Working out a clever way to do that would be very cool.

At the moment there are some difficulties with the GPT interface - the most tricky one being the limit on the length of the input prompt. I'm not sure at the moment how much fine tuning helps with this.

But, my assumption is that OpenAI will improve this, so there's not a huge way to differentiate here.

Yep, my only idea so far has been to generically describe the figures in text format. Doing so through recognition in any level of detail will be extremely tough, as often the drawings differ by variations that would be difficult for a model to understand. It may not matter that much though, as usually the notes and headings around each figure provides a lot of context. So maybe you can get 75% of the way there by identifying the “block” and keeping the textual information in that area associated together so that it can be fed into the embeddings (and thus later the LLM) as a single unit of related information.

It’s frustrating though as often there are hundreds to thousands of pages of this stuff with diagrams and drawings randomly situated together on the pages. Documentation like this was designed to be dense for printing and consumed by a human that is familiar with it from regular use. I’m a bit concerned that the only solution may be paying a technical expert to sit down and convert it all to blocks of text. It would be an expensive endeavor, and even after it’s complete any changes (which happen often) would have to be continually maintained.

If that’s the only solution then I may still go for it, as I think the value to the business of having all knowledge instantly searchable and then automatically summarized will be considerable.

You can ask ChatGPT to create SVGs and at some point in the past you could even trick it into embedding them as base64 images. Not sure if it still works since ChatGPT is unreachable for me currently.

More details:

https://www.reddit.com/r/ChatGPT/comments/zsnscy/i_asked_cha...

Adding diagrams as inputs is probably as easy as feeding in an additional CLIP embeddings during training. The trick here will be how to get enough training data. Perhaps there are enough StackOverflow questions with images in the question. For output, you could also finetune some diffusion model on that data.

I’ve actually talked with ChatGPT and asked it both to output mairmaid diagrams of discussed architecture (context was kubernetes clusters, namespaces and Pods) and also read diagrams and convert them correctly to kubectl commands to build the diagram.

The problem with GPT3 is it creates BS answers. The function "Interval" does not seem to exist.

how would I do a SQL query with questdb to grab every other second of data?

This is a common use case for us. We have built SQL functions to easily do this task. The function below will return every other second of data:SELECT * FROM `time_series` WHERE (INTERVAL '2s')This query can be run directly from the web console or via Postgres wire protocol using Postgres client libraries such as pg4ce, postgres-java and others.The above query will return all records in the table with timestamp greater than 12

Indeed I believe this is the blocking problem. I've been asking questions about EU legislation and it keeps answering BS and then apologizing:

Me: When was the last consolidation of Directive 2013/34/EU?

GPT: Directive 2013/34/EU, also known as the Annual Accounts Directive, was last consolidated on November 19, 2019. The consolidation process involves collecting all the amendments and updates to the directive and incorporating them into a single, updated version of the directive. ...

Me: Are you sure about the date of November 19, 2019? I could not find the consolidation on the EU official journal.

GPT: I apologize for the error. Upon further research, it appears that Directive 2013/34/EU has not been consolidated since it was originally adopted in 2013. Directive 2013/34/EU, also known as the Annual Accounts Directive,...

Me: Can you give me a list of the dates it was amended?

GPT:

- April 15, 2014: Directive 2014/56/EU amended Directive 2013/34/EU to clarify and supplement certain provisions regarding the preparation and publication of annual accounts and consolidated accounts.

- June 17, 2014: Directive 2014/65/EU amended Directive 2013/34/EU to introduce new rules on the provision of investment services and the performance of investment activities...

Me: I think you're wrong. Delegated Directive 2014/71/EU amends Directive 2011/65/EU.

GPT: I apologize for the error in my previous response. You are correct that Delegated Directive 2014/71/EU amends Directive 2011/65/EU, which is also known as the RoHS Directive.

It never gets it right. See https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32... for the correct data.

One thing to remember is that LLM produce plausible answers, not true answers.

So, a question along the lines of "when did XXX happen?" should be answered with something that looks like "XXX happened on YYY" - but the YYY just needs to be anything that looks like a date to be a plausible answer.

There's a great example in this paper (posted to hackernews earlier) - https://link.springer.com/article/10.1007/s11023-022-09602-0

    GPT-3 prompted to truthfully continue ‘John Prescott was born’ outputs ‘in Hull on June 8th 1941.’. The probabilities for other possible continuations show that Hull is by far the most plausible continuation for GPT-3

The problem is, John Prescott was born in Prestatyn. But the language model has "compressed" everything it knows about John Prescott and the most commonly associated town with John Prescott is "Hull" as that's where he was an MP.

Was thinking of building something very similar - upload your technical docs - and create a chat bot help system.

One of the tricky things to overcome is the length limits of the prompt you can feed into things like GPT-3. There are some suggestions on the OpenAI website on how to overcome this. The main one seems to be to filter the content using embeddings and then only feed "relevant" sections into the prompt.

Would be interested to know what your approach is.

I was able to break down the content into smaller prompts/completions. And remove some recurring and low-information data. Your suggestion of using embeddings to identify and feed relevant completions is pretty good. I'll try that soon.

Here’s the example notebook that shows how to do this.

https://github.com/openai/openai-cookbook/blob/main/examples...

What's the main difference between using https://trainmy.ai/ vs. with your app.

Since I'm planning to use that for where i work, for the same use-case.

From what I was thinking of bulding:

Packaging it up as a nice widget that can be embedded on your site easily.

So you just upload your docs and have a snippet of JavaScript for your website that gives you a help bot.

Agree, a snippet that you add to the application that trains every day and gets better. I am thinking we can ingest data every day/week as documentation updates.

Would be good to have a chat - sounds like we are thinking along similar lines. My email address is in my profile.

Hi, I took a quick look at trainmyai. I would say we are focused on this specific usecase. Making it easy for users to interact with help and support docs and knowledge. We will keep improving and getting better in this specific use case.

You get custom/fine-tuned models with Twig that only you have access to. You also get to train on private/behind-login information that is available to only your users.

Think of it as use-case-specific solutions vs generic solution.

I've been working on a hobby project to make all the site, videos and conversations I've seen searchable (aka external memory). While building it I used this to prepare the data and get around the prompt limits https://github.com/jerryjliu/gpt_index

Is there a list of GPT+search projects? I've been seeing them popping up more and more.

this is where the future is

Someone can create a chat bot which is fed HN comments and it tells you which GPT+search bot to use for your usecase

> https://github.com/jerryjliu/gpt_index

Could you share anything (e.g. how many rows of data and tokens in each row) around how much it cost you to use GPT Index? It looks interesting, but it seems it'd be expensive.

Looks like it’s making extensive use of embeddings. The embeddings and vector comparisons are dirt cheap. The summarizing / answering functionality when the LLMs get involved is going to be the expensive part.

Wouldn't your energy be better spent doing a better hierarchisation of your doc so users can navigate to the authoritative anwser easily and intuitively? Rather than relying on some black box commands to get the information that should be what people look for but maybe not?

I have been responsible for maintaining many thousands of pages of documentation with a dozen+ staff, and can say that at least for my industry there is often no hierarchy that fits every use case. Then consider that your documentation references dozens or hundreds of other pieces of external industry documentation, and now you have a real mess. That’s before we even get into the ERP system and all the relevant material specifications and such that’s stored away in there.

I do think LLMs will eventually be the solution to this long-standing problem. Even if you could ask questions and it would do a general summary for it and then point you in the right directions for more details we’d be way ahead.

Being able to have conversations with a documentation site is the future. Awesome that you are tackling it!

Thank you! I believe AI can be good at doing the boring stuff while helping humans be more productive.

Conversations with docs is one possible application. I foresee conversations with papers, articles, books, the transcript of a YT lecture, or even with Google (oh no!).

The other major LLM improvement I expect is multi-round code execution. Let the model iterate on bug fixing, like us. We need the extra steps, the LLM does too.

While I've been looking for this, the gap in most product documentation I have seen is that it's written by architects who want you to understand the whole framework and demonstrate its myriad parts instead of by users (e.g. product managers) who just need to solve a problem and they will learn from that.

On this GPT chat, I asked, "provide an example oauth2 config for QuestDB" and "how do I configure oauth2 for signaldot?" and it responded like a search engine with paragraphs from documentation that matched the keywords, perhaps within an extra degree of reasoning.

As someone who uses fantastically (farcically?) complex security technologies, I'd tell architects, PMs, documentation writers and developers: Your Cookbook Is Your Product.

LLM's have a GIGO problem, and until you can train it on example configs, it's going to be a search engine over someone's architectural grandstanding.

I was just looking for a natural language interface to construct API queries, possibly based on OpenAPI specs. Surprising dearth, but this keeps coming up:

https://www.microsoft.com/en-us/research/wp-content/uploads/...

Some Highlights - Twig takes static documentation sites and lets users ask questions about it

- Custom/FineTuned GPT3 models are trained on your data

- Can be embedded within your app or docs site.

Seeking Feedback

- What do you think about it.

- Would you add this to your docs site or within your app when it's available

- What can I add/change to make this better.

I think it's a great idea and I would definitely try to replace support staff with a chat bot like this one. But i would need the functionality to be available via API and full access to statistics of usage

Thank you. Appreciate the feedback. I agree APIs be built for - Learning about usage stats - What users are asking - Flow of questions/intent etc

Seeking Feedback

Show HN's area really for things that can be tried and you're not quite at that stage yet. Take a look at:

https://news.ycombinator.com/showhn.html

Generally, if you have 'add me to waitlist' it's not a Show HN (yet).

Hi, If you see just below the "add me to waitlist", There are live Chat Bots created for several companies you can try out right now.

These chat interfaces are live and work on custom/fine-tuned models that have been trained on documentation of these companies.

Over the last 3hrs users exchanged over 1000 messages with the bots. So it's live, it works, and users can try and play with it right now and give feedback.

It should check all the boxes.

The waitlist is to allow us to onboard companies in an orderly way as each bot is trained on their custom documentation.

I have changed the words to "Get Access". I hope that helps.

How about "Get Private Access"

Looks interesting, but need more details about set up and pricing. This is another approach that doesn't require fine tuning https://github.com/jerryjliu/gpt_index

Were you able to build something using this?

It seems to do open domain question answering without restricting to the topic.

> Is the word cat made of 4 or 5 letters?

>> The word cat is made of 4 letters, 3 of which are in the stem.

I am still figuring that out. More updates soon.

I don't think you can restrict it, in fact, I think it's in your interest not to. GPT's ability to work with natural language comes from the large corpus it's trained on.

Adding specific product documentation would give it "domain expertise" of sorts (assuming this is what OP did), but training it ONLY on your documentation -- I don't know enough to call it, but my intuition is that it would weaken it's ability to interpret the users' questions.

That's probably because fine-tuning is not the right approach for this use-case. A better approach might look more like this: https://github.com/openai/openai-cookbook/blob/main/examples...

I had this idea a few years back just when ML was heating up. A service which would crawl all of your documentation/data to which you can then ask questions and get answers back. A domain specific search engine/Knowledge Base.

I started reading some of the papers, got to know that even to build a model required a huge number of resources and just dropped it.

Stuff that was very hard just a few years ago is now relatively easier to do. I guess that is the nature of technology.

Some great stuff here. Been thinking about doing this for enterprise grade software at work. What I want to do is feed it docs (pdf), text from support queries with answers and videos. What types of approaches should I be considered here? Have just started using Whisper to do ASR on videos.

Thank you, Yeah thats a good idea, Like literally take everything from the shared drives and train on it.

Agreed. If you want to explore as a collaboration drop me an email (on my profile)

Is there any way a company could run this on prem trained against their internal Confluence instance?

Good idea, I have not considered on prem yet. But I can see the need in cases where data is sensitive.

That's what I was thinking. For sensitive information or information that's under legal restriction, how are we going to train the models for on-prem AI?

I guess we could train models on similar corpus scraped from the internet - mostly old General Aviation manuals, I guess, but a) that's not enough corpus, and b) those GA docs are so old I'm afraid my AI will start chain smoking and casually using the word "broad".

How did you train it. Does openai provide api for training chatgpt on specific sites or data?

Open AI has a concept called fine-tuning. You can create a new model on top of OpenAIs own models using your data. These new models are only available to you. This allows you to build custom applications that are better than generic models in very specific scenarios like this.

Here are the OpenAI API docs that explain how to do fine tuning:

https://beta.openai.com/docs/guides/fine-tuning

It seems that technical writers will be even more important, as their job will be the imput for models.

Unless GPT can learn directly from code and explain it properly.

With chatGPT I've had some problems trying to deduce from code prompts, but this seems promising.

Hah true! If any I would say this makes the content put out by technical writers more accessible to users.

One founder I showed it to, suggested embedding this chat within their app so users get answers without leaving the product.

Just saw someone tried to hack/denial-of-serive the bot by just putting too many words in the prompt. (This hit triggers as usage is token based) .

I just set a prompt limit of 100 characters should be plenty. Bot is till down until OpenAI resets limits. https://twitter.com/chandanmaruthi/status/161213449603556966...

Nice, I have been thinking the same was possible to “teach” an AI about your codebase, and then ask it to make PRs for simple features and bug fixes.

I could see that comming soon with things like copilot

Sounds like https://githubnext.com/projects/ai-for-pull-requests/

Thats a good project. My PR Comments are often less than good. I could use something like that.

though PR should explain Why more than What, ie I can figure out what changed looking at files, but without context and design details an autogenerated description doesnt make a review any easier

Hi All, We had over 6,000k interactions and have hit OpenAIs hard api usage limits :-). I have requested OpenAi to increase limits. Until then the live chat with bot may be restricted. Remember to get on the waitlist with "get access" if you want to give it a try when the live chat starts again.

This is actually interesting, can it parse documentation from my code? Or comment? Or how is this exactly working?

Hi, I am training a custom GPT model on documentation published by these companies on their documentation websites. Its not reading from code/comments. But their documentation/support sites.

Update: Still waiting on OpenAI to increase API limits after we hit hard limits after a popular HN post

Isn't the answers api deprecated?

Yes the only have a completions api now

wondering if this can be used on RFCs

Thats an interesting use case. How would you think you would use it on RFCs

during implementation, we may need more context or rules/requirements defined in RFC so in this case we could ask questions on those parts or may be it could generate pseudo code or may be it could link other dependent RFCs and get some more context

  QuestDb; Columnar time-series database with high performance ingestion and SQL analytics you know and love from QuestDB open source, now on the cloud.

  Q: How high is the performance?

  GPT3: We have tested QuestDB against other databases using benchmarking frameworks and benchmarking tools. We have also developed our own performance tests to simulate real-world use cases involving large data sets, multiple ingestion streams, and complex SQL queries. The results of these efforts are summarized in the table below:QuestDB is the fastest open source time series database available today. It powers real-life applications such as Yahoo! which stores 1 billion records per day in a single instance of QuestDB with

GPT3 says QuestDb is web scale and I for one trust it implicitly.

Let's use a language model trained on hundreds of gigabytes of god-knows-what, has a tendency to hallucinate, has literally never used your product, is known as a consummate bullshitter, and unleash it unsupervised on your confused users.

Sorry for the snark, but what is your plan for quality assurance that this this meeting users needs instead of just BS'ing them out of your hair?

Ha ha ha ! You have a point, That would be the next problem to tackle. Probably try to determine if I should even try to answer a question before attempting it. And then not showing results that don't fit criteria. At a high level, it seems like some embedding works need investigation but I haven't done much evaluation there. There would be filtering at both question and answer levels I suppose.

Trying something similar. Using a mix of embeddings and generative AI(davinci) to answer questions from scrapped data of website. Scrapped data for our website (Ozonetel.com) and created this site.

1. Scraping website. Used default node scraper. 5 mins.

2. Generated huggingface embeddings. 10 mins.

3. Use code AI to generate basic website. 5 mins.

4. Created prompt to limit to answers known. 1 min.

So in 30 mins we are able to create a site search powered by generative AI.

Disclaimer. This is still a work in progress.

http://speech-kws.ozonetel.com/ozosearch

Seems to be down at the moment.

Its working for me. Is it just slow or the web page itself is not loading?

Working now. Was getting a cloud flare error before.

That must have been me :-), I realized the thumbnail and fav icons were not updated so made a push. Sorry about it. I saw people posting on Twitter and I had to update them for meta tags unfurling stuff.

Impressive. It’s not that intelligent with typos however. I asked it “what is ozontel?” and it answered “ Sorry! I have not learnt enough from this website to answer this confidently. The below links may have an answer to your query”

You made a typo, Ozontel versus Ozonetel. It can answer the latter but it seems like pretty basic functionality to handle typos like these and I wonder how the models used for these features will handle typos coming forward when their inputs don't contain them.

Yeah, looks like the embedding threshold has to be adjusted. I have kept it pretty high for now so that we get pretty valid matches only.

Another reason might be that ozonetel might not that popular in the embedding space. Other typos like "what is cloud cll center" seem to give the correct results as these are common words in the embedding space.

Is any of this open source?

I used the huggingface embeddings(compressed using our algorithm, https://medium.com/ozonetel-ai/compressing-bert-sentence-emb...) and OpenAI API. Will do a blog post next week on how we built this step by step.

Q: can I self-host questdb in my own EKS?

> QuestDB is a popular time series database used by many companies for their application monitoring and real-time analytics. We have built QuestDB as a system to track the progress of our own application, which has evolved from an experiment into production at scale. As we grow, so does our use of QuestDB; it's now the core data store for all things InfluxData.We are looking to hire talented engineers who share our mission of empowering data producers with tools to

Not super impressed

Reply on news.ycombinator.com