I feel like this is pretty hard to believe, to be honest. How on Earth, given all that's happened last year, could we possibly not at least be "heading in the right direction" the way things are?
> My personal view is this: If you view the current generation of LLMs as pioneering efforts in a long-term research program, then the results are encouraging. But if you consider this technology as an immediate replacement for hand-coded software, it’s quite hopeless. We’re nowhere near the necessary level of reliability.
I think especially the overview of the studies provided in the article are valuable. You can draw your own conclusions from the results.
Generative transformers only address part of the problem space, and perhaps I should have written “appear to address”. It is unlikely that they are much of a step along the path — perhaps, to use a car analogy, they are a crankshaft.
Perhaps a better analogy is the analogue computers of the 40s-60s and the later digital/analogue hybrids. Computationally interesting for the problems of the age but ultimately a dead end.
The LLM models that are all the hype are superb at things like copy writing and the more traditional humanities but struggle at logic. With a year of reasearch we can reach human parity in things like psychotherapy but for something that requires logical reasoning rather then repeating bits of text it's not that clear.