427 karma · joined August 30, 2019
To be honest, this might be very challenging. BUT it may be possible. Just try entry level Analysis or Linear algebra. They often start kind of from scratch. How to construct numbers and so on. You could go very slow but at least you would be ready for real math...
https://en.wikipedia.org/wiki/Criticism_of_Human_Rights_Watc...
I understand why it is so expensive, since it is not a mass product. Or in other words, it should be 10x cheaper, if it would be a mass product.
Anyway, to bring it to the next level, how big should it be? Maybe 10T? 100T?
By the way the AI scientists get a lot of respect and admiration see Ilya for example.
What I meant is, most likely assuming that you are using pytorch / jax you could code down the model pretty fast. Just compare it to llama, sure it is far behind, but the llama model is under 1000 lines of code and pretty good.
There is tons of work, for the training, infra, preparing the data and so on. That would result guess in millions lines of code. But the core ideas and the model are likely thin I would argue. So that is my point.
The dataset is more challenging, but here msft can help - since they have bing and github as well. So they might be able to make few shortcuts here.
The most time consuming part is compute, but here again msft has the compute.
Will they beat chat-gpt 4 in a year? Guess no. But they will come very close to it and maybe it would not matter that much if you focus on the product.
In contrast, Instagram Reels are annoying; even after reporting certain content, the same or similar Reels reappear. We don't even need to discuss TikTok. From my perspective, it tends to have far more hateful content.
To me it is unclear what does it say about much more complex models? So for example there might be already so much structure, that you do not need to generalize out of sample and maybe humans don't do it as well.
On the personal level, maybe you are right - but it doesn't matter, if he is right...
- the theory is not mainstream, guess it is not attractive enough to study it right now
- the theory is not able to make any new predictions (yet). This has to change I think to get traction.