The "learning" part has always caused me grief. Always.
Unfortunately it's beyond recovery. But "hallucination" and other new coined terms.. they definitely have problems.
Unfortunately it's beyond recovery. But "hallucination" and other new coined terms.. they definitely have problems.
It's as if we called programming "magic" and programmers "wizards" and then kids ask "is Harry potter real" and somebody says "yes"
It's not hallucinating. It's an analogy. But analogies are not definitional. We're going to wind up in conversations about AGI and use of imprecise analogistic language will alter how people discuss this problem.
Sorry you think I'm irratinal. I think you're being obtuse and not considering real world consequences of the ontologies used outside of the field.
A more difficult exercise is coming up with apt names without anthropomorphising--something I rarely succeed at. I'm often drawn to analogs in the musical realm (Scales, Sequences, Signals, Tuning), carpentry (Blueprints, Components, Scaffolding), libraries (Resources, Pages, Indices), the natural world (Roots, Trees, Seeds, Leaves), physical media (Slots, Packages), publishing (Versions, Releases) and even cooking (Mixers, Recipes).
We often understand technology by way of analogy or prior experiences. Whenever dealing with the abstract or unknown, metaphors are often the first tool to make sense of them [0].
They learn patterns in data automatically. These are not explicitly programmed by humans.
The objection is that the word 'learn' is generally used in reference to how humans learn. There is a lot implied in this usage, and a long history of use by the general populace. By overloading it as 'things that are kinda like human learning, in some ways' you must ignore swathes of that implicit meaning to remain correct while using it. It is no longer a phenomena, but a class of phenomena.
The problem they pose is that people aren't going to do that - they'll continue using 'learn' as a stand-in for 'human learning' even when discussing ml adjacent topics, and that is damaging to effective discourse.
Practically speaking, that ship has sailed. Regardless, it's a reasonable take.
You can literally see their implied scenario play out in this thread - they claim that llms are doing machine-like learning rather than human-like learning (an entirely uncontroversial claim, humans are almost certainly not transformers in disguise) and the conditioned response is along the lines of "you're wrong, they do learn! obviously you don't understand the most trivial aspects of machine learning!"
This exchange merely proves their point (which is really the observation): overloading this word in particular causes grief. Misunderstandings abound and arguments arise from no actual disagreement in object level reality.
Whoever doesn't understand the term are non-experts. It's not an issue among people who studied and research the topic.
No they thought humans are creating and tweaking the models manually. Fundamental misunderstanding.
I fail to see how this is not literally the case.
The 'logic behind the weighting methods' refers to something like cross-entropy loss minimization via AdamW (or insert optimizer / loss function of choice) over some dataset. I read that as 'the process that decides the weights'.
If the model is unaligned ("too racist"), fine tuning with rlhf or alternative of choice is a modulation of the aforementioned 'logic behind the weighting methods'. It is a modification to the process that is deciding weights.
Judging by your response, you read 'logic behind the weighting methods' as 'the weights'. I don't think this is a reasonable interpretation unless you're trying to construct a strawman to argue against.
Sanity check via chatgpt:
> In the context of generative pretrained transformers (GPT), "logic behind the weighting methods" refers to the rationale or reasoning behind the techniques used to assign different weights to words or tokens during the model's training process.
- https://chat.openai.com/share/8af4ce45-2c4a-485a-ae8a-e1e016...
So now we are talking about two separate things. The algorithm used to train the model, and the model itself. They did not alter the algorithm that generated the model and they also did not directly alter the model itself. The neural net was modified indirectly through training. That's machine learning 101. When fine tuning a model we do not alter the training algorithm that generates the model. As far as we know, no one is manually tweaking weights or manually editing nodes in its neural network (yet). That is far beyond current knowledge and abilities but is an active field of research in the very early stages.
The training process is a search in model space where bad answers are penalized and good answers are rewarded. The algorithm itself doesn't need altering to find a "less racist" model. There's nothing special about racism that requires a fundamental change in the transformer architecture. It's just language and semantics. They trained the shit out of it by RLHF (Reinforcement Learning from Human Feedback) until it was less racist.
If you only look at the very high level superficial details, they could be said to be similar. That is all. The details differ in nontrivial ways.
> They did not alter the algorithm that generated the model and they also did not directly alter the model itself. The neural net was modified indirectly through training.
It is beyond contesting that multiple, different, algorithms are used to train SOTA language models. RLHF exists because pretraining on a broad corpus for predictive loss minimization, then instruction tuning on specialized datasets, is insufficient in various ways (alignment, "racism", etc).
RLHF and pretraining/finetuning use different algorithms. They are similar in some ways - they can both incorporate backprop with a stateful optimizer like AdamW. This does not make them the same.
RLHF computes loss differently from how it is done in pretraining and finetuning. That, alone, constitutes a modification of the logic behind the weighting process.
Once again, the logic behind the weighting process refers to the training algorithm. Not the transformer architecture. Not the weights themselves. The process that decides what the weights are.
> The training process is a search in model space where bad answers are penalized and good answers are rewarded. The algorithm itself doesn't need altering to find a "less racist" model. There's nothing special about racism that requires a fundamental change in the transformer architecture.
Neither I nor the op claimed that anyone changed the transformer architecture to make it less racist. See my last reply w.r.t. reinterpretation of claims in service of strawman construction.
> They trained the shit out of it by RLHF (Reinforcement Learning from Human Feedback) until it was less racist.
As per above, RLHF is a nontrivially different process to what was used to train the too racist model in the first place.
You are confusing yourself and still not quite understanding the basic concept.
Machine learning has three parts:
1. Training dataset
2. Learning algorithm.
3. The model.
The data is input to the learning algorithm which generates the model.
To make ChatGPT less "racist" they provided new training data and rewarded it for the correct answers.
The process or algorithm of RHLF was not modified specially to handle "racism". These are general learning algorithms. They simply added new training data. This is basic Machine Learning 101.
- [0]: https://arxiv.org/abs/2005.14165
- [1]: https://arxiv.org/abs/2307.09288
For example, from the llama2 paper, section "3.2 Reinforcement Learning with Human Feedback (RLHF)":
> RLHF is a model training procedure that is applied to a fine-tuned language model to further align model behavior with human preferences and instruction following. We collect data that represents empirically sampled human preferences, whereby human annotators select which of two model outputs they prefer. This human feedback is subsequently used to train a reward model, which learns patterns in the preferences of the human annotators and can then automate preference decisions.
The rest of 3.2 should be enlightening as well (in service to understanding how RLHF is actually quite different from the training algorithms used in pretraining or finetuning).
> The data is input to the learning algorithm which generates the model.
> To make ChatGPT less "racist" they provided new training data and rewarded it for the correct answers.
For frontier language models, this is strictly incorrect. Multiple rounds of different training algorithms are used, each with its own objectives. RLHF is more complex than just 'put better data into same old pretraining algorithm.'
> The process or algorithm of RHLF was not modified specially to handle "racism". These are general learning algorithms. They simply added new training data.
Nobody is claiming RLHF was modified, but that the 'logic behind the weighting methods' are modified. Pretraining, then finetuning on specialized dataset does not produce a sufficiently aligned (non-racist, gender-neutral, etc) model, no matter how much extra data you throw at it: the answer to this is the introduction of a different algorithm, RLHF, which more accurately fits the model to desired alignment than unsupervised autoregressive loss minimization does.
The sequence of events here is:
1. unsupervised pretraining on mixed dataset
eval result: "it's just continuing the prompt instead of answering questions or being useful"
researcher response: "let's put more focused data in to make it respond better"
2. instruction tuning on task-specific dataset
eval result: "it's answering questions now but it has inherited biases from the pretraining corpus (it's too racist, sexist, or otherwise not aligned with desired values)"
researcher response: "we've hit the limit on what you can reasonably accomplish with this training algorithm, let's try something else" (see GPT3, section "5. Limitations")
3. rlhf on human preference dataset
eval result: "well, it's less racist now"
> You are confusing yourself and still not quite understanding the basic concept.
Once again, as per my last reply: "If you only look at the very high level superficial details, they could be said to be similar. That is all. The details differ in nontrivial ways."
The basic ontology that you're trying to argue exists, is counterfactual. It boils down to: "put data into training algorithm, get model out. if it's too racist, put aligned data in and get non-racist model out." It's not hard to understand, you're just wrong. SOTA models are not produced this way. Full stop. The details of training phases are nontrivially different both algorithmically, and in the datasets used.
At this point we've exhausted all reasonable discourse; it's clear that you're not engaging in good faith and I remain convinced that your position falls apart under the most trivial application of rigorous analysis. If you have any actual source to back up your claims, I'd love to see it. Beyond that, good day.
FFS. That's exactly what I have been trying to explain to you. RHLF is the general method. So you agree with me. I am fully aware that there are different algorithms used at different stages of the training process. No one is hardcoding racism out of the model or hardcoding antiracism in the training algorithm. That is learned automatically. It's still the same basic machine learning 101 concept.
> At this point we've exhausted all reasonable discourse; it's clear that you're not engaging in good faith and I remain convinced that your position falls apart under the most trivial application of rigorous analysis. If you have any actual source to back up your claims, I'd love to see it. Beyond that, good day.
Totally unnecessary and uncalled for. You have repeatedly accused me of bad faith which lowers the discussion to a childish level of vindictive garbage. You were not making your point clear before and I was making the effort to respond sincerely to that, trying explain what I thought you misunderstood. It seems that you are still confused and are arguing my point for me without realizing it. Thanks for making my point clearer and adding interesting technical references to back it up. You have provided the sources to back up my claims. Cheers. Have a good one.
As per my last reply, I don't believe further debate of this point would be productive. We've both made our arguments. I'm comfortable letting mine stand on their own merits.
If you'd like to start a meta-discussion about this discussion, and any grievances therein, I'll happily participate if we can keep it civil. I think understanding why and how debates can breakdown at the limits (and what those limits are) provides real value in avoiding such breakdowns in the future.
> Thanks for making my point clearer and adding interesting technical references to back it up. You have provided the sources to back up my claims.
I am glad to have made a positive impact on you in some small way, at least.
> Totally unnecessary and uncalled for. You have repeatedly accused me of bad faith which lowers the discussion to a childish level of vindictive garbage. You were not making your point clear before and I was making the effort to respond sincerely to that, trying explain what I thought you misunderstood. It seems that you are still confused and are arguing my point for me without realizing it.
In the spirit of the above, if you are being sincere: I must genuinely urge you to consider why someone might believe your arguments to be unsatisfactory, and how charitable it is not to only consider incoherent formulations of the arguments of others. Put simply: if you see someone's argument as trivially unsound, is it possible you have misunderstood their position?
If you're admitting that you did not understand my argument with sufficient clarity to contest it, on what grounds can you claim to know what I've misunderstood in the making of it? My claim follows from this line of reasoning. You are, necessarily, by your own admission, arguing against a strawman.