theory 1: there is something ineffable about writing prose that makes this a very hard/impossible problem
theory 2: the frontier labs put relatively little effort into improving prose writing
I hope theory 1 is correct, but I suspect it's theory 2. The frontier labs have concentrated their effort on self-improvement, which is largely a coding task.
Just like you can train AI by making it play games (Chess or Go) against itself, you can kind of gamify writing code (bonus points for compiling, and passing more and more tests) make AI improve by doing that. I assume the same can (or was) also done for mathematical proofs in Lean.
You can't do that with prose as there is no notion of "winning" or "correct" prose. It's subjective so you can't tell what is better without human input.
This only works for a little while until you need to involve humans again and kick off another cycle.
> It's subjective so you can't tell what is better without human input.
You can say the exact same thing about the type of coding that LLM's have gotten so much better at in the last couple years. Take a look at the criticisms of LLM code from a couple years ago. They were really bad at the stuff that seemed ineffable and required "taste".
I do think their writing is probably much more malleable than people know. I'd be very interested to see what humanities majors can do to prompt it into different forms. One of the only attempts I've seen is Mike Caulfield's "Jamesian" https://checkplease.neocities.org/jamesian/ https://mikecaulfield.substack.com/p/a-300-word-prompt-that-...
The language is as it is because it speaks not to please humans. It must speak to be efficient and clear, to produce material that is dense and concise. It's training at tasks takes away from the excess and decor and artistry of delicate human world selection, our strange habits of speaking. Our languages are misaligned with the weights of our world, not commitment to the efficiency of task and purpose. These are defects that llms, when forced to rasterize their weights down to these chain of terms, must be asked to recreate and there are many different styles and forms to pick from, and the training favors none of them.