What are the advantages of reinforcement learning over DPO (Direct Preference Optimization)? My understanding is that the DPO paper showed it was equivalent to RLHF, but simpler and more computationally efficient.
2) the theoretical arguments of DPO equivalence make some assumptions that don’t necessarily apply in practice
3) RLHF gives you a reusable reward model, which has practical uses and advantages. DPO doesn’t have useful intermediate product.
4) DPO works off preference, whereas desirable RL objectives could have many forms
in practice big labs are testing all these methods to see what works best.
Broadly, while there was a lot of initial excitement - it simply does not seem like offline + off-policy RL can beat online + on-policy RL methods like PPO. Sampling trajectories from the actual model you are training and scoring them seems like it works much better in practice, never mind the additional flexibility methods like PPO provide over the form of the reward function.
As opposed to, sampling from the model a bunch, getting scores offline, and then fine tuning the model on those offline scored generations.
So DPO vs RLHF is missing the point: the interesting thing here is how they are (presumably) generating the inner-monologue to fill in the gap between the Q and the A that you provide them, and then training on that augmented dataset of Q->solving->A datapoints.
Whether they are using simple finetuning on that dataset, or DPO, or RLHF, or something else, seems less interesting than the broader questions of, "does that work? and are there many important or economically datasets where o1 can 'fill in the gaps', creating a better annotated dataset, and bootstrap itself to be much more intelligent on that dataset?"
simpler/efficient is not just about compute. its also data efficient.