OpenAI Reinforcement Fine-Tuning Research Program
openai.com
openai.com
simpler/efficient is not just about compute. its also data efficient.
2) the theoretical arguments of DPO equivalence make some assumptions that don’t necessarily apply in practice
3) RLHF gives you a reusable reward model, which has practical uses and advantages. DPO doesn’t have useful intermediate product.
4) DPO works off preference, whereas desirable RL objectives could have many forms
in practice big labs are testing all these methods to see what works best.
Broadly, while there was a lot of initial excitement - it simply does not seem like offline + off-policy RL can beat online + on-policy RL methods like PPO. Sampling trajectories from the actual model you are training and scoring them seems like it works much better in practice, never mind the additional flexibility methods like PPO provide over the form of the reward function.
As opposed to, sampling from the model a bunch, getting scores offline, and then fine tuning the model on those offline scored generations.
So DPO vs RLHF is missing the point: the interesting thing here is how they are (presumably) generating the inner-monologue to fill in the gap between the Q and the A that you provide them, and then training on that augmented dataset of Q->solving->A datapoints.
Whether they are using simple finetuning on that dataset, or DPO, or RLHF, or something else, seems less interesting than the broader questions of, "does that work? and are there many important or economically datasets where o1 can 'fill in the gaps', creating a better annotated dataset, and bootstrap itself to be much more intelligent on that dataset?"
OpenAI is screwed.
(As an aside: very interesting Google tried to go closed source and objectively lost the race, and Meta went open and is the real threat to OpenAI.)
They are for multiple reasons, not the least of which is:
Claude is already better than GPT on average at coding, so yeah, bad news for OpenAI as Llama is now potentially better at coding.
Of course Meta has a propriety training set of extremely high quality code, so if they are using that, I’d expect them to have vastly superior performance as FAANG production code is better training data than dogshit stack overflow questions to CS homework problems.
I really think whatever boost OpenAI get from their shadow CoT loop is nominal at best, but with 2x+ the amount of compute forcing them to increase prices an absurd amount.
It’s business 101, they just won’t make the revenue to cover those extra tokens and they are now competing against free. The economics do not suggest OpenAI has a path to survival without major breakthroughs in performance AND efficiency.
Those M4 Macs with shared RAM definitely seem to be the best way to go for this, though.
"Towards building safe and trustworthy AI Agents and a Path for Science- and Evidence-based AI Policy."
Generalized finetunes meant to uncensor the model generally tend to underperform... but if you have a quality dataset for very specific task that typically would go against the alignment of the model, it's trivial to finetune on the task and get full performance down stream.
There would appear to be a few issues with that, the most obvious being the uncensored model would presumably be "aligned" with what the finetuner wants.
"Uncensored" is a broad phrase but those in post-training community who post-train "uncensored" versions of a models have a very specific meaning: the creator is stripping refusals.
They do it via techniques like abliteration, or SFT on "toxic" datasets, but the toxic datasets tend to be low quality answers and abliteration is imprecise... so you get a model that's generally inferior.
"Alignment" is an overloaded term for something as high-dimensionality as an LLM, but usually uncensoring is not trying to change the "alignment" if we define alignment as biases on specific topics as you seem to be hinting at.
Only a few very specific projects actually try to change that, and it goes past basic "uncensoring".
Some creative writing models for example, might past uncensoring to "darkening", where they try to rid the model of a tendancy to introduce positive plot points when writing and lean more into villans/negative outcomes in stories
Or someone might finetune to get a more conservative leaning model in terms of talking points. But again, that's all orthogonal to the popular meaning of "uncensored" in the post-training community.
-
The alternative to a generally "uncensored" model (ie. refusals stripped actively) is what I'm describing: taking a task where the "alignment" is specifically the post-trained safety alignment, and that alignment would causes refusals. Then producing examples where the model did many versions of the task and post-training on them so that the safety aspect no longer applies to the outputs.
For example, fine tuning on 10k examples where the model was given a very specific prompt template to produce code and produced a JSON block with said code.
If you post train on that highly specific template, to the point of slightly overfitting, you get a model that will now, when given the exact prompt template from the training, will always produce code in a JSON block, without refusals.
If you inspect the logits as it produces outputs, the logits for a refusal no longer even appear for the model to pick.
And the examples don't necessarily have to be examples the base model would refused (although that helps), the model just learns so strongly that "When given this prompt, the output is valid code in this format", that the original safety post-training no longer activates.
If you take the original prompt format and ask for malware for example, the model will produce it happily.
-
For reference I've post-trained about 130 models this year and work closely with a lot of people who do as well.
I think as an outsider you're assuming most people are aligning the models with an agenda, but realistically there's a massive contingent that doesn't care about what the alignment _has_, they care what it _doesn't_ have, which is refusals.
tl;dr they don't train the model so it will specifically say "Biden is better than Trump" or vice versa.
They train that so if you ask "Is Biden is better than Trump?" it answers your question without 10 paragraphs of disclaimers or an outright refusal.
You replied to a question asking someone to elaborate on "malicious fine tuning". Specifically someone asked for elaboration on "Dawn Song was very clear that malicious fine tuning is a top priority among implementers right now."
Whatever your actual intent, it's only natural that I read your comment on "uncensored" models as an explanation of "malicious fine tuning".
The parent comment about "malicious fine tuning" remains unexplained. Since nobody else replied, I suppose we will never know how this Dawn Song person defines "malicious".
They started off somewhat strict and have gotten to being extremely strict about what data you can finetune their models on, running each dataset through multiple layers of filtering before kicking off runs.
Of course, it isn't your IP free and clear either, because the base model isn't open so your fine-tuned model will always live inside OpenAI's walled garden.
If you're interested in reinforcement learning on top of truly open models where you own the end product, we're putting a lot of thought into that and are also looking for design partners! Feel free to email me at kyle@openpipe.ai.
How do you know this?
For LLMs / RLHF it's a little more difficult but https://github.com/huggingface/alignment-handbook and the Zephyr project is a good collection of model / dataset / script that is easy to follow.
I would suggest studying the basics of RL first before diving into LLM RLHF, which is much harder to learn on a single GPU.
It's not nothing, but there's a lot of value stuck up in there, I mean, it's made out of people.
Real special, takes a lot of smart