Comment by bob1029 | Hacker News Reader

Comment by bob1029 | Hacker News Reader

I feel like there is an emerging consensus that [Chat]GPT 3.5/4 is not just 1 big model.

A large part of the magic in the final product appears to be many intermediate layers of classification that select the appropriate LLM/method to query. The cheaper models (e.g. Ada/Babbage) could be used for this purpose. Think about why offensive ChatGPT prompts are rejected so quickly compared to legitimate asks for code.

Imagine the architectural advantage of a big switch statement over models trained in different domains or initial vectors. Cross-domain queries could be managed mostly across turns of conversation w/ summarization. Think about the Stanford Alpaca parse analysis diagram [0]. You could have an instruction-following model per initial keyword. All of "Write..." might fit into a much smaller model if isolated. This stuff could be partitioned in ways that turn out to be mildly intuitive to a layman.

Retraining 7B parameters vs 175B is a big delta. The economics of this must have forced a more modular architecture at scale. Consider why ChatGPT is so cheap. Surely, they figured out a way to break down one big box into smaller ones.

  [0]: https://github.com/tatsu-lab/stanford_alpaca/blob/main/assets/parse_analysis.png

> Imagine the architectural advantage of a big switch statement over models trained in different domains or initial vectors.

Given that the emergent abilities come from the large parameter count and massive amount of training data, using smaller models seems like a distinct disadvantage.

> Given that the emergent abilities come from the large parameter count

Where can I find evidence of this?

https://openreview.net/pdf?id=yzkSU5zdwD

https://arxiv.org/pdf/2203.15556.pdf

There were also some informal comparisons of GPT models with various parameter counts.

Excellent info - I did find a bit in the conclusion from the arXiv article:

> While the desire to train these mega-models has led to substantial engineering innovation, we hypothesize that the race to train larger and larger models is resulting in models that are substantially underperforming compared to what could be achieved with the same compute budget.

This mirrors some of my experience. Training/tuning a 7B parameter model feels like goldilocks right now. We are thinking more about 1 specific domain with 3-4 highly-targeted tasks. Do we need 175B+ parameters for that? I can't imagine it would make our lives easier at the moment. Iteration times & cost are a really big factor right now. Being able to go 10x faster/cheaper makes it worth trying to encourage the smaller model(s) to fit the use case.

https://ai.googleblog.com/2022/11/characterizing-emergent-ph...

https://arxiv.org/pdf/2303.12712.pdf

That doesn't mean that if your parameter count is big enough to handle the input, the next step can't more efficiently hand off [parts of] the query to a specialised module. The emergent properties of LLMs learning maths and chess from text inputs alone are interesting, but specialised engines are much more efficient and more accurate.

I agree. But I think do think they use a tiered approach: they have an uncensored model at the very bottom, to which the queries go to first. And then there's the "clean" model, and (for lack of a better explanation) the two outputs are then "XOR"d to get the clean public version.

I agree the core models are uncensored (as this would introduce a lot of noise), but I don't think any queries are allowed to touch them before a cheaper model first performs screening. If someone is spamming toilet humor, there is no reason to exercise the full 175B+ parameter model each time.

The only reason ChatGPT is affordable is because of caching, filtering bad prompts, etc. The underlying LLM would be far too expensive if everyone was hitting it 1:1.

Agree 100%. Have you noticed any lag with more "difficult" queries?

Yes. I think there are multiple levels of LLM for each use case now.

The stuff that absolutely must run on 100B+ models can be pre-classified by something like Ada/Babbage. Attempts at arithmetic or dimensional analysis can be binned and shipped to a monster model. Anything that is more routine information retrieval needs maybe goes to a 7B parameter model.

Lots of models with lots of temperature levels/hyperparameters/etc is the only way to achieve the kinds of performance we are seeing. The secret sauce is looking increasingly like a huge classification layer.

Nothing you see in ChatGPT is as it appears. Lengthy conversations are managed with recursive summarization. Every conversational turn is potentially handled by a different LLM.

Not exactly the same, but Google’s Pathways architecture is sorta along the same lines

https://blog.google/technology/ai/introducing-pathways-next-...

I suspect they must be selecting different models based on the question asked. I've noticed this myself asking simple questions vs difficult questions, chatGPT is much slower on the difficult questions even though each token has the same computation cost if given to the same model

Reply on news.ycombinator.com