427 karma · joined August 30, 2019
Why do you think that Llama can be replaced? I mean it is extremely costly to train that thing. And it is there even a clean open source data set for the task?
PS: wouldn't be surprised if Meta, OpenAi, or google will train something for a Billion $ in costs of compute or more.
PS: Not an expert just speculating.
Anyway just curious do you or someone else have more sources in this format?
At least this way it is very helpfull to think about the methods. Kind of agree that the formular does look very very similar to well known methods. But on the other hand, the author doesn't explain the transformers in a similar simple way since it is not obvious why you could stack kernels and get better results.
But the most difficult part is: - intuition behind them, why they work at all. What a good parameter set is - how to check if training goes well - what to do when things go wrong - how to train large models. I mean gpt4 was trained for a long time. For sure they did some things before and during the long training period.
And so on. I mean because the models are rather „simple“, I think the best engineers get paid pretty high, because it is not that much about the models but everything around it.
*if they can execute
I Would be surprised if that is the case.
Let R+ denote the set of positive real numbers. Find all functions f : R+ → R+ such that for each x ∈ R+, there is exactly one y ∈ R+ satisfying xf(y) + yf(x) <= 2.
It just struggeld to reason. So I would be very surprised if the plugin somehow helps here.
People will use the best solution. Chrome came after firefox and ie and opera and become more populare because it was better.
I think Hossenfelder described it in the video the problematic in Chaos: The real problem with quantum mechanics
The point was made by Sabine Hossenfelder