i think o1 isworse than o4 for coding.
Your experience aligns with what I've seen: 4o and o1 are better at filling gaps from each other where there should be an upward trend of intelligence instead of having to jump around as we do currently.
So the real question is: are these models forgetting their skills, do we need to make them larger? Does distillation and generalisation work?
So many questions that might end up as LLMs being unable to escape the problem of catastrophic forgetfulness.