3,869 karma · joined May 8, 2009
To give you an idea:
Tesla revenue: 74.86B
Toyota revenue: 260.13B
Tesla market cap: 498.56B
Toyota market cap: 234.96B
That's roughly a 7.4x difference in expectation vs. reality.
* A fixed (but large) vocabulary of sub-word like tokens as inputs.
* Attention mechanism for learning the correlation of words in a fixed sequence window.
* Implementing this attention mechanism in the form of matrix multiplies rather than some other complex math - it allows it to be parallelized and run on GPUs fast.
* Having enough layers of these encoders to have a huge amount of parameter space. ~175B parameters in the case of ChatGPT.
* Feed the model a lot of data - in this case, pretty much the entire internet as text.
* Self supervised learning: we take sentences from the internet and mask out some words and force the network to predict the missing word. Turns out this works extremely well. We don't use the traditional supervised learning inputs -> (prediction, label) paradigm that was the standard 10 years ago.
* RLHF (Reinforcement learning from human feedback). Take generated text from GPT-3 (the underlying generative model) and ask humans to rate different completions. Retrain the model from those ratings.
* A massive compute infrastructure that is able to train this model in a reasonable amount of time, allowing for iteration on hyperparameters. For example, what's the optimal attention head-size? How many encoder layers are good? What should the sequence length be? What should the embedding dimension be? etc. In OpenAI's case, they used thousands of GPUs and thousands of CPUs provided by Microsoft/Azure.
In summary, relatively simple model, parallelizable on GPUs, trained on a lot of data.
To add to your comment, Google has been using BERT to power Search since 2019: https://blog.google/products/search/search-language-understa...
I'm going to guess that the only reason they don't use larger models is because of the compute cost. ChatGPT at 4 billion users with today's hardware is an unsustainable business. However, that thought leads me to imagining: if Google offered Search "Premium" using the latest LLMs, how much would people pay for it?
sudo dscacheutil -flushcache;sudo killall -HUP mDNSResponder
An entire area of software that is completely unnecessary, but because the gods at Google gifted it to us, people have to pretend it's amazing
Plenty of reasons
The fact that most software doesn't continue to function even on the same platform, and on the same hardware, is a massive indictment of the software industry's standard practices.