Hacker News Reader

Comment by hooande | Hacker News Reader

> "Bing is going to have to come up with some kind of radical revenue sharing too if they want anything written after 2023."

ChatGPT doesn't include anything written after 2021. I certainly wouldn't use it to find an air fryer. The results will be from over a year ago. I would want to see what the newest air fryer options are and it would be really important to have to up to date pricing.

AFAIK there is not a way to update a large language model in real time. You have to train on the entire dataset to do a meaningful update, just like with most forms of neural networks. For ChatGPT that takes days and costs hundreds of thousands of dollars. Every time.

It's great for explanations of concepts, and programming, and a few other things. But with the huge caveat that all of the information you're looking at is from one year ago and may have changed in that time. This really limits the utility of ChatGPT for me.

View on HN

OpenAI is already working on solving this

https://openai.com/blog/webgpt/

Neat! I've seen so many discussions of the cost of continually retraining ChatGPT with new knowledge (and the energy efficiency of that, etc.) but had a similar thought that you can probably use a GPT-like approach to do "next word prediction" for a command-based web crawler to gather up to date data and then use the GPT-we-already-have to combine/integrate found content using the classic next word prediction.

Sometimes I feel that what makes humans cool is that we (well, some of us!) have good internal scoring on when we lack knowledge and must go in search of it which makes us go down different branches of next-action-in-line.

Someone pointed out that the energy cost of training GPT is roughly on par with a single transcontinental flight. If so, I don't think this is a limiting factor in any meaningful sense - you could spend that much energy daily, and it would still be a drop in the bucket overall for any moderately large business.

I would imagine trying new datasets on daily basis wouldn’t be trivial ?

The bottleneck would be the number of workers on sites like mechanical turk available to create the datasets. Might take a few more years before amazon and facebook get enough third world countries to the point they can exploit their labour online to create daily training sets.

That’s a very solvable problem though. If Microsoft decides to integrate ChatGPT with bing, they have the resources to retrain the model on a more recent data set, and even do it somewhat regularly

You don't even need to retrain if you use retrieval transformers. That is the real revolution waiting to happen. Deepmind already unlocked it with RETRO, but I don't know why a public version hasn't been released - hooked into the live internet.

OpenAI have webgpt too https://openai.com/blog/webgpt/

Reply on news.ycombinator.com