But more importantly, I don’t have to solve any problems with languages that are elaborate practical jokes, so I’m not worried about the implications of an LLMs ability to be useful.
But more importantly, I don’t have to solve any problems with languages that are elaborate practical jokes, so I’m not worried about the implications of an LLMs ability to be useful.
It's not as if the models here were asked to write a kernel in Brainfuck; the medium tier of problems here contains such apparently insurmountable tasks as "calculate the nth prime".
The thing about LLMs is there doesn't seem to be a way to teach them genuine reasoning. You can spend a month teaching an LLM brainfuck and it would likely still fail at a novel problem. Whereas if a human studied brainfuck for a month they would probably be quite competent at a novel problem
This is just being needlessly dismissive. Esolangs are (and have been) an area of active CS research for decades. I know I'm a bit of an esolang nerd, and while some are jokes, most focus on specific paradigms (e.g. Piet is visual, bf is a Turing tarpit, etc.).
> I think most Python programmers when tasked with “now do it in brainfuck” would fail.
This is untrue. Given internet-level awareness and infinite time, virtually all developers should be able to go from Python to brainfuck (trivially, I might add.) Did you even look at the test sets? It's all pretty basic stuff (palindromes, array traversal, etc.—we aren't using pandas here). I mean, sure, it would take forever and be mega annoying, but manipulating a head and some tape is hardly difficult.