Hacker News Reader

Comment by epups | Hacker News Reader

There is not enough written text to scale a model like ChatGPT at even 10x its current parameters, so improvements will have to come from optimizations and/or longer training (at diminishing returns).

View on HN

Enough data wont really be a problem. https://us.sganalytics.com/blog/2-5-quintillion-bytes-of-dat...

I wouldn't be surprised if we create more data in one year than is in the entire internet from the start

quintillion bytes of social media probably have very limited value.

Really? All of humanity imparting everything they think and know publicly seems incredibly valuable. 100% serious.

If you're looking to make a tweet generator, sure. If you're looking to harness the power of facts, no.

The main issue might be that the training set may get polluted by a lot of AI generated text (content that is not supposed to be there, but is there because AI-generated content is becoming widespread online).

Storage is cheap. With ChatGPT being the preminent system, and with OpenAI recording all of its output, it should be possible to exclude modified answers from the training data.

So interesting that people think facts are the only thing that's valuable. Your comment isn't a fact, does that mean it's worthless?

When looked at individually it's not worthless, but when you are talking about a corpus that significantly lacks facts, then the corpus itself is worthless for any mainstream usage other than a social network text generator.

How do you know that? There are thousands upon thousands of books that haven't been digitized.

GPT-4 Will Have 100 Trillion Parameters — 500x the Size of GPT-3

That's likely a massive over-estimation. I would bet it's around 1 trillion.

That is just an internet rumor and officially denied by OpenAI.

Reply on news.ycombinator.com